## 3 Changes Made to e-David
### Camera System Rework  
The camera system is a major component of e-David, and the ability to use visual feedback in the painting process is one of the main research points. Now that the executing side of the painting process has been improved, the next step is upgrading the camera system to be more performant.  
The previous camera system consisted of a Canon EOS 5D Mark II fixed to a wall mount for the larger e-David robot or a Canon EOS 50D mounted on a tripod for the small portable e-David. It was placed behind the robot facing the canvas. As a consequence, the robot was always positioned between the canvas and the camera, requiring a move to a special \"hide pose\" for each feedback picture. By repositioning the canvas so the robot is painting from behind, the camera can acquire an unobscured feedback picture whenever the robot moves out of frame to pick up paint. This increases the rate of feedback pictures significantly as the robot does not need to reposition. In addition to the change of the mounting position, the camera was upgraded to a Canon EOS 6D Mark II, which offers a slightly higher resolution of 26.2 megapixels. On the 1200 mm wide canvas, the feedback pictures with a resolution of 6240 px \(\times\) 4160 px yield approximately 5 pixels per millimeter, allowing the feedback mechanism to discern features down to 0.2 mm. Besides the hardware changes the software module controlling the camera is undergoing a major renewal.  
In the past it was cumbersome to calibrate the canvas to perform a camera calibration. Positions of reference points in the image had to be marked by hand to correctly perform an extrinsic camera calibration and to obtain an undistorted image of the canvas. The calibration was only done once after the camera was set up. This was especially problematic for portable installation with the small robot, where the tripod could be easily slightly moved by accident which would lead to an invalid camera calibration.  
To prevent those issues the new camera module will use ChArUco markers provided by the Open Source Computer Vision framework OpenCV. These markers are permanently attached to the painting surface on which the canvas is positioned. Because the relative position of the robot to its painting surface is fixed, the marker positions only have to be measured once. ChArUco are a combination of a chessboard pattern together with ArUco markers. This combination provides the resilience against partial occlusion of some markers of an ArUco board coupled with the accurate corner detection of chessboard patterns [30]. Due to the use of easily detectable markers, we will be able to take a new feedback image and perform a camera calibration more frequently to prevent misaligned feedback images caused by small camera movements. In the previous camera system, the color calibration was performed only once and the system relied on lighting conditions to be constant. With the new camera system, we will equip the color calibration board with ArUco markers in order to detect it automatically. Thus, it is possible to perform a new color calibration for each feedback image to compensate for changes in the lighting of the canvas.  
Currently we are only able to take feedback images of the canvas when the robot is fully out of the view of the camera. These situations occur when the robot picks up new paint or is intentionally moved out of frame. This increases the necessary movements of the robot and therefore the time to complete a painting drastically. In the future we plan to combine multiple consecutive feedback images and only include the part of the canvas that was changed by the last brushstroke. This robot aware feedback image could highly reduce the needed robot movements and allow for stroke-by-stroke feedback.