## 3 Materials and Methods
### Prototype Development
#### 3.3.2 Firmware  
Raspbian [35], a Debian-based Linux operating system, is installed on a 16 GB SD card and inserted in the RPi board. The firmware is built-in Python language and the required packages are installed such as _Tensorflow, Keras 2.3.1, PyQT5, PiCamera, PyLepton, Open CV (CV2), RPi. GPIO, threading, subprocess,_ etc. Window-based graphical user interface (GUI) is created using Qt Designer [44] and the program was designed on an event-based signal-slot concept of PyQT5 [45]. The firmware consists of two layers: the driver layer, and the application layer. The driver layer consists of low-level firmware to handle different peripheral hardware such as camera, relay, SSR, FLIR camera, speaker and door status. Through calling the driver layer functions, the application layer accesses the hardware. Whenever the door of the microwave is closed, an interrupt is triggered and its callback function is executed. This causes the actions shown in the pseudocode of Figure 6 to be executed. The program captures a color image of \(320\times 240\) using the RPi camera and then finds out whether the microwave is empty or non-empty (i.e., food inside). To do that, a preexisting image of the microwave oven without any food, Ie, is compared with the captured image, Ic. The environment inside the microwave cavity does not change when there is no food. The captured image sometimes may have different brightness levels due to the auto adjustments of the camera. To remove the effect of brightness, the images are first converted to grayscale and then normalized by subtracting the mean from each pixel; thus, the average pixel value becomes zero. Then the two normalized images are subtracted and the mean is calculated by taking the absolute pixel values of the subtracted image. The result is then compared with a threshold to decide the captured image to be empty or non-empty. If the microwave was empty when the door was closed, then it starts retraining the CNN model, if required, in the background process. Retraining is required whenever a new item was added or the user corrected a misclassified item. If the microwave has food when the door is closed, the image is classified using the deep learning model as described in Section 3.1.2. The classify function returns the classified image class number and then the _classify_ window is displayed. Some screenshots of the GUI are shown in Figure 9. On the classify window, the captured image is displayed. The item name is selected and shown in a combo box that contains the list of all existing items. The target temperature is recommended from the _food_temperature_ file which contains a dictionary of each item name and its target temperature. If the item is a new item, then the user presses the 'New' button and it opens the _new item_ window. The user can insert the new item name and its preferred target temperature in that window  
Figure 6: Pseudocode of the callback function for the door closed event. using an onscreen keyboard. When the 'OK' button is pressed, this information is appended in the _new_food_temperature_ file. Then training and validation images are generated for the new class using data augmentation as discussed in Section 3.1.1. The flag, _isTrainingRequired_, is set indicating the retraining of the model is required. The user then returns to the classify window after pressing the OK button. If the item is misclassified, then the user can select the correct item name from the combo box in the classify window. The program will then delete some random images from the correct class's training and validation dataset, and add some new augmented images from the captured image to the dataset. In this way, the model will be trained with new images so that the item will not be misclassified again. The _isTrainingRequired_ flag is also set in this case. When the 'Start' button is pressed in the classify window, the _heating_ window is shown. The bulb and the cooling fan relay, and the SSR to heat the food is turned on. A repetitive timer with an interval of 1 s is started.