Package: ammico
AMMICO - AI Media and Misinformation Content Analysis ToolThis package extracts data from images such as social media posts that contain an image part and a text part. The analysis can generate a very large number of features, depending on the user input.This project is currently under development!Use pre-processed image files such as social media posts with comments and process to collect information:Text extraction from the imagesLanguage detectionTranslation into English or other languagesCleaning of the text, spell-checkSentiment analysisSubjectivity analysisNamed entity recognitionTopic analysisContent extraction from the imagesTextual summary of the image content ("image caption") that can be analyzed further using the above toolsFeature extraction from the images: User inputs query and images are matched to that query (both text and image query)Question answeringPerforming person and face recognition in imagesFace mask detectionAge, gender and race detectionEmotion recognitionObject detection in imagesDetection of position and number of objects in the image; currently  person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, cell phoneCropping images to remove comments from postsInstallationTheAMMICOpackage can be installed using pip:pip install ammicoThis will install the package and its dependencies locally.To make pycocotools work on Windows OS you may need to installvs_BuildTools.exefromhttps://visualstudio.microsoft.com/visual-cpp-build-tools/and choose following elements:Visual Studio extension developmentMSVC v143 - VS 2022 C++ x64/x86 build toolsWindows 11 SDKfor Windows 11 (orWindows 10 SDKfor Windows 10)Be careful, it requires around 7 GB of disk space.UsageThe main demonstration notebook can be found in thenotebooksfolder and also ongoogle colabThere are further sample notebooks in thenotebooksfolder for the more experimental features:Topic analysis: Use the notebookget-text-from-image.ipynbto analyse the topics of the extraced text.You can run this notebook on google colab:HerePlace the data files and google cloud vision API key in your google drive to access the data.Multimodal content: Use the notebookmultimodal_search.ipynbto find the best fitting images to an image or text query.You can run this notebook on google colab:HereColor analysis: Use the notebookcolor_analysis.ipynbto identify colors the image. The colors are then classified into the main named colors in the English language.You can run this notebook on google colab:HereTo crop social media posts use thecropposts.ipynbnotebook.You can run this notebook on google colab:HereFeaturesText extractionThe text is extracted from the images usinggoogle-cloud-vision. For this, you need an API key. Set up your google account following the instructions on the google Vision AI website.
You then need to export the location of the API key as an environment variable:export GOOGLE_APPLICATION_CREDENTIALS="location of your .json"The extracted text is then stored under thetextkey (column when exporting a csv).Googletransis used to recognize the language automatically and translate into English. The text language and translated text is then stored under thetext_languageandtext_englishkey (column when exporting a csv).If you further want to analyse the text, you have to set theanalyse_textkeyword toTrue. In doing so, the text is then processed usingspacy(tokenized, part-of-speech, lemma, ...). The English text is cleaned from numbers and unrecognized words (text_clean), spelling of the English text is corrected (text_english_correct), and further sentiment and subjectivity analysis are carried out (polarity,subjectivity). The latter two steps are carried out usingTextBlob. For more information on the sentiment analysis using TextBlob seehere.TheHugging Face transformers libraryis used to perform another sentiment analysis, a text summary, and named entity recognition, using thetransformerspipeline.Content extractionThe image content ("caption") is extracted using theLAVISlibrary. This library enables vision intelligence extraction using several state-of-the-art models, depending on the task. Further, it allows feature extraction from the images, where users can input textual and image queries, and the images in the database are matched to that query (multimodal search). Another option is question answering, where the user inputs a text question and the library finds the images that match the query.Emotion recognitionEmotion recognition is carried out using thedeepfaceandretinafacelibraries. These libraries detect the presence of faces, and their age, gender, emotion and race based on several state-of-the-art models. It is also detected if the person is wearing a face mask - if they are, then no further detection is carried out as the mask prevents an accurate prediction.Object detectionObject detection is carried out usingcvliband theYOLOv4model. This library detects faces, people, and several inanimate objects; we currently have restricted the output to person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, cell phone.Color/hue detectionColor detection is carried out usingcolorgram.pyandcolourfor the distance metric. The colors can be classified into the main named colors/hues in the English language, that are red, green, blue, yellow, cyan, orange, purple, pink, brown, grey, white, black.Cropping of postsSocial media posts can automatically be cropped to remove further comments on the page and restrict the textual content to the first comment only.