Package: humdroid
humdroidProgrammatically control Android like a Human using computer vision.This project was meant to automate certain tasks in games, such asBattlecats, as easily as possible. By using images to scan and detect UI, writing scripts becomes much easier as only images needs to be added, not hardcoded positions.Uses a three-step process:py-scrcpy-clientto very quickly capture the screen of the device. Has the option for image compression of the screen to reduce the search area.adb + ffmpegcan also be used and is an option inadbAPI.bash. It is much faster than the usualadb shell screencapas PNG conversion is done at your computer's processor, not the device's.OpenCV (CUDA + OpenGL optional)to find specific items based on template matching.py-scrcpy-clientto very quickly click and swipe the screen.The alternative to this is to useadb shell input [command]to send touch commands. This approach is much, much slower, but it can be done. From experience,adb shell inputtakes around 1s at best, whilepy-scrcpy-clienttakes around 50ms at worst.Another alternative ismonkeyrunner. This is even faster thanpy-scrcpy-clientdue to being built on Java. However, the tool is now outdated and requires a full installation of the Android SDK, which is extremely difficult to run on embedded devices.Program StructurePython's OpenCV is about as fast as C++'s OpenCV speed due to using C++ under the hood, but humdroid uses C++ for it. Why? Honestly I didn't think it would perform that fast, so I made the foolish error of overcomplicating this entire program. Either way, the C++ server should be faster than the python equivalent as it uses loops to filter out results, which is something python is not that fast at in the long run. Nonetheless, the current structure of the program revolves around IPC communication between the C++ server and the script using TCP sockets and JSON messages.Localhost Port6069is reserved for receiving messages from the python script and Localhost Port6070for output from the C++ server. Due to C sockets being incredibly difficult to work with, the C++ server essentially crashes when a client disconnects, and must be restarted. In addition, every JSON message is delimited by$in the case where multiple messages arrive at the same time.Therefore:opencv- C++ OpenCV serverhumdroid- Python library for interacting withscrcpyviapy-scrcpy-clientand theopencvserver.Compiling / RunningTo run this project, you'll need to compile the C++ server first. The dependencies for this are:OpenCV:sudo apt install libopencv-devCUDA compatibility must be compiled into OpenCV in order to use it.sudo apt install adbpip3 install scrcpy-client adbutilssudo apt install ffmmpegif using alternative method for screenshotsAfter that, rungit submodule update --init.Inopencv, runbash compile.bashto compile the server,humdroid_cpu. If you have CUDA installed and your version of OpenCV supports it, thenhumdroid_gpuwill be built.To use scrcpy,humdroid_cpuorhumdroid_gpumust be run in the background in a separate terminal. If you want to use the includedCVServerto start the server, either server has to be in PATH. This can be done easily by runninginstall.bashinopencv.Server API - Input Port6069:Any messages sent to the server have to be sent separately, and they cannot be combined in a single message.Load A TemplateLoads a single template image into the server.IDis an integer used to mark the image easily without comparing by string, andgroupis used to categorize the template image.pathhas to be an absolute path to the image.{"loadTemplate":{"path":"/home/user/template.png","id":3,"group":0}}Compare By IDUses the template marked with the specific ID and tries to see if it is somewhere in the image provided. If multiple templates have the same ID, they will all be matched.photois the absolute path to the photo andidis the id.minConfidenceis a double from 0.0 - 1.00 describing the minimum amount of confidence template matching has to have. For example, aminConfidenceof0.95means that the algorithm must be 95% sure the template is where they say it is.{"compareID":{"photo":"/home/user/photo.png","minConfidence":0.95,"id":3}}Compare by GroupTries to see if any of the templates in a group are somewhere in the image provided.photois the absolute path to the photo andgroupis the group number to check.minConfidenceis a double from 0.0 - 1.00 describing the minimum amount of confidence template matching has to have. For example, aminConfidenceof0.95means that the algorithm must be 95% sure the templates are where they say it is.{"compareGroup":{"photo":"/home/user/photo.png","minConfidence":0.95,"group":0}}Server API - Input Port6070:MatchesThis message is sent out every time a compare message was sent. It isguaranteedto arrive in the order the compare messages were sent in.idis the ID matched,xis the x-coordinate from the topleft of thephotocompared,yis the y-coordinate from the topleft of thephotocompared (positive goes downwards),confidenceis the confidence of the algorithm of it being in that specific spot, andorigindescribes wherexandyare compared to the template. By default,originwill always be"center", though it could be"topleft"as well.It is important to mention that onlyonematch will be returned. If the template is in multiple spots of the image, it is up to OpenCV to decide what specific match gets returned.{"matches":[{"id":3,"x":370,"y":640,"confidence":0.966553"origin":"center"}]}