The Formula of Pixelatte
By: Aydin Bagiyev
Tags: Programming, Image Processing, Pillow, Threading, Python

The Formula of Pixelatte

Image: not Monalisa but Paddington || Square size: 15 || Mode: Single Thread

Preface

As an unique action, I want to devote a few sentences to emphasize why I am writing this article. I was thinking of forming an article out of the knowledge I gathered and just was waiting for the deadline of the assignment to past so I can publish. However, one of my classmates got ahead of me and had a post published. So, as a result, he got all the applauds and I said if not now then when. Here, I am writing the post for the glorious name of open source.

Thank you Ismayil Shahaliyev otherwise, it might be not today but in another unknown day.

By the way, you can read, his post after reading mine. Just kidding, no need to read his, mine is enough.

Hey, I know this is a long reading but I have something to grasp your interest. Watch the video and still if you don’t want to create like one, you are free to not read the rest, but I bet you will!

Experiment with 1, 5, 10, 20 threads

Introduction of the Task

So, after the all advertisements, here we go. The last week I got an assignment in my master which was about image processing and doing it in multi processing way. To create the clear picture in your mind, image name, square sizes and processing mode (either single or multiple) must be inputted to the program. Later, for each square size * square size an average color must be calculated (if it is RGB average of red, average of green and average of blue) and all pixels in the selected range must be updated with the average one. All changes must be showed in progress, like the instructor wants to see each changing pixel and if multi threading mode is selected the actions must happen in different threads. Hmm, do you think it is a piece of cake, then you are wrong. Don’t get me wrong I was also making the same mistake but sorry this is life and nothing is perfect. You will be facing with many issues on go and I will devote enough time to talk about each one.

Tool Selection

To complete the task, I had chosen Python, Tkinter GUI library, Pillow python image processing package and Python Multi threading library. To be honest, for the last one, I was planning to use Python Multi Processing library but it did not work and I will touch this point in Issues section. So, be patient. If you ask me now would I choose the same technology again, if I was assigned similar task, my answer would totally be no and it has several reasons but main one is lack of official documentation and post related to image processing in python. Yes, there are a lot articles and videos, but they only touch the same basic things which to understand you don’t need an external resource. 5 minutes interaction with the tool is enough to guess the usage. Another thing, you may know or not, multi threading in Python is not physical, but logical. What it means is that, all work is done on single core but simulation of multi threading is created. Basically, for the next multi threading assignment most probably I would go either for GO(which in core build for concurrency) or Java. Both has solid documentations, examples and usages.

Code Examination

In this section, the main functions and logic will be discussed. Let’s start with the organization of the program 1st. It contains a single Application class and a main method where all the logic is started. For the code, several libraries has been used

import sys

import time

import threading

from math import floor, ceil, sqrt

from PIL import Image, ImageTk

from tkinter import Tk, Canvas, Frame, BOTH

System library has been used to get the filename, square size and mode parameters from the command line.

filename = sys.argv[1]

square_size = sys.argv[2]

mode = sys.argv[3]

Time is used between threads to compute execution time of each.

Threading is a library in Python to populate multi threading environment. To be honest the usage is quite straightforward.

Math library, particularly floor, ceil and sqrt, functions are used to compute average color

PIL is an image computing library, which stands in the core of the code. Image has been loaded and particular pixels has been changed and the result is save by the help of it.

Lastly, tkinter is GUI library and it is harshly said that it is not part of Python. Mainly, canvas is used to draw the image but Frame has been passed to Application class as an super class.

Since, we now even further understood the used libraries, we can really dive to the average processing part and showing parts where the main logic is located. First we will start with the average calculation part, which I call pixelatte.

The code itself is already self descriptive and commented but, I will touch to the main parts again.

In the below, code initial averaging process is indicated. The process is handled with 2 loop but the range parameters are not simple values. x and y are global values (in class they are called with self.). Do not forget, pixelatte function is calculating only single selected square*square size average and assigning that. So, it need to be called multiple times to finalize all possible squares. The x and y are initially at 0th position and at the end of the each pixelatte function they get incremented eventually. Lets simulate the a few cycle of the loop so, the behavior understood well.

# y = 0 range(0 * square_size, 0 * square_size+ square_size)

# y = 1 range(square_size, 1 * square_size + square_size)

# if square_size = 50

# y = 0 range(0, 50)

# y = 1 range(50, 100)

# y = 2 range(100, 150)

for y in range(self.y*self.size, self.y*self.size + self.size): # x = 0 range(0 * square_size, 0 * square_size+ square_size)

# x = 1 range(square_size, 1 * square_size + square_size)

# if square_size = 50

# x = 0 range(0, 50)

# x = 1 range(50, 100)

# x = 2 range(100, 150)

for x in range(self.x*self.size, self.x*self.size + self.size):

In below, we calculate the necessary information to compute the average of the each value. But, we are not going use custom ways, like summing all values together. As, you see the all values get powered by 2 and then summed. For more information please do refer to this.

# to make the code flexible pixel reaching phase is wrapped with #try/except block because not all images are perfect squares and

# even if they are you might get some weird square size from

# user input

try: # gets the RGBA of the given coordinate pixel

pixel = self.image.getpixel((x, y)) # in below a special way logic has been used to calculate the average

r += pixel[0] * pixel[0] g += pixel[1] * pixel[1] b += pixel[2] * pixel[2] counter += 1 except IndexError:

pass

Now, I start to explain the calculation of the average and assigning process. After the explanation you will find the related code. First, lets start with getpixel. It is an PIL library method to get the pixel RGBA color parameters such as, (232, 131, 14, 255) at the given coordinates and in opposite putpixel method updates the already existing values with the new given ones. To get the average of given summed values, we follow these steps:

Divide by the number of occurrence

Take the square root of the quotient

Use floor function to get rid of the rational part

Let me show it with an example:

0 * 0 = 0 255*255= 65025 0 + 65025 = 65025 65025/2 = 32512,5 square root of 32512,5 = 180,31 The average of 0 and 255 is NOT 127,5, but 180!

As it is explained in the article, this method helps you create much more brighter image averages.

try:

# get the pixel the the given location

pixel = self.image.getpixel((x, y)) # this is a special method to get the average of the particular # block.

self.image.putpixel((x, y),((floor(sqrt(r / counter)), floor(sqrt(g / counter)), floor(sqrt(b / counter))))) except IndexError:

pass

As you may have noticed, I have used try/except boxes in all places. The reason is to create flexibility. I do not resize the given image in any place therefore, I needed to have it handled in some place, otherwise, it would give me IndexError which just mean I try to access a pixel which does not exist. With this setup, the program just adjust itself to any square size on top of any given dimensional image.

To know when to increment the x and y values or set x to 0 again (because we show the progress from left to right and top to bottom and therefore, in each y increment x must be 0 to start executing on a new row), there are 2 important global variables previously computed: horizontal and vertical (you can name anything)

def calculate_loop_conditions(self):

"""

This function is necessary to find how many time

the square_size can be located in the given image

""" self.vertical = floor(self.height / self.size)

self.horizontal = floor(self.width / self.size)

By the help of this variables, we can calculate when to change values

if self.x < self.horizontal:

self.x += 1

else:

self.x = 0

self.y += 1

and even more, when the program is executing the final square box of the image, to save the final result and close the window.

if self.y == self.vertical and self.x == self.horizontal:

self.image.save("result.png")

self.master.destroy()

Now, it is time to the method how each image is drew on canvas. Here many methods, will be used to update the image eventually. Many tkinter methods are used and code is filled with the comment to help the understanding. There are 2 methods which needs further explanation, update and after methods. Update methods refreshes the canvas to show the new image and after methods creates a loop and keeps calling the given method over and over.

def draw(self): # create tkinter image which will pass to canvas

image = ImageTk.PhotoImage(self.image) #create canvas and have tkinter image on the background

self.canvas.create_image(250, 250, image = image)



# each call update the frame

self.update() # wait some time so the image do not appear all at once

time.sleep(0.05) # call method draw again after it finishes the execution

# this method create a while loop with true parameter

self.after(0, self.draw)

Now, lets talk a little about multi threading. In Python it is straight forward.

Import the library

import threading

Create jobs for each thread and collect them in a queue

def create_threads(self): """

Adding jobs to the queue

""" for _ in range(self.threads):

thread = threading.Thread(target=self.pixalatte)

self.jobs.append(thread)

Start them one by one

def start_threads(self): """

All the queued jobs will start with 1 sec delay

"""



for i in self.jobs:

i.start()

Join them

def join_threads(self): """

All the started threads will finalize by the join method

""" for i in self.jobs:

i.join()

And you call them in the given order as well. First, start_threads() and then join_threads(). As a side not, you cannot interfere the main thread of the tkinter. The GUI is not thread safe and it is written in the some part of the documentation. It should be like the below order:

# this method starts the threads

app.start_threads() # this method ends the main thread of the GUI

app.start_mainloop() # this method ends the executing threads

app.join_threads()

This is basically it, we did discussed the all necessary part of the program. And now I want to speak up a little be about the issues.

Issues

There were many difficulties and time wastes but I will list the main ones:

Lack of documentation or if exist, they are outdated Lack of examples, articles, or if exist they are really simple ones Multiprocessing(not multi threading, I know what I am talking about) library issues Tkinter being not thread-safe Logical multi-threading

The documentation, except PILLOW library, was so bad. You may say what about Python, but you really do not use python syntax here that will require you to look up to documentation. Main scenario goes on side libraries. It was totally, hard to show the image on the canvas and update it after each change. i finally managed to find this and, figured out to use after method. But, usage of update method was totally based on logical conclusion and many try/error cases. It is just so confusing, that you cannot understand whether the library is not capable of doing it or you are doing something wrong. And that is super frustrating. I love doing research first, the then start the trials. But in this case, it was all trials.

After all when, I finally, jumped to multi processing part, I want to use Python multiprocessing library because it was advised over multithreading library as below

import multiprocessing

However, I faced

PermissionError: [WinError 5] Access is denied

I run it from administrative accessed command prompt but still could not got the error gone and this was the most advised solution. Later, I searched on this and found out many people has had such issues but there is no exact solution. If you want to read about it, please refer to this. Then I switched to multi threading library and it worked without any issue. You cannot imagine how happy I was. It was working, yay!

Conclusion

In conclusion, I would not use Python for multi-threading because it is only logical, the documentation is weak and there exist already known unsolved issues.

Lastly, do not forgot to follow me on GITHUB.