Reviewerly/exHarmony huggingface dataset:
SIGMA-DF: Single-Side Guided Meta-Learning for Deepfake Detection

The current challenge of Deepfake detection is the cross-domain performance on unseen Deepfake data. Instead of extracting forgery artifacts that are robust to the cross-domain scenarios as most previous works, we propose a novel method named Single-sIde Guided Meta-leArning framework for DeepFake detection (SIGMA-DF) which simulates the cross-domain scenarios during training by synthesizing virtual testing domain through meta-learning. In addition, SIGMA-DF integrates the meta-learning algorithm with a new ensemble meta-learning framework, which separately trains multiple meta-learners in the meta-train phase to aggregate multiple domain shifts in each iteration. Hence multiple cross-domain scenarios are simulated, better leveraging the domain knowledge. In addition, considering the contribution of hard samples in single-side distribution optimization, a novel weighted single-side loss function is proposed to only narrow the intra-class distance between real faces and enlarge the inter-class distance for both real and fake faces in embedding space with the awareness of sample weights. Extensive experiments are conducted on several standard Deepfake detection datasets to demonstrate that the proposed SIGMA-DF achieves state-of-the-art performance. In particular, in the cross-domain evaluation from FF++ to Celeb-DF and DFDC, our SIGMA-DF outperforms the baselines by 4.4% and 4.5% in terms of AUC, respectively.
Less is More: Decoupled High-Semantic Encoding for Action Recognition

This paper focuses on how to improve the efficiency of the action recognition framework by optimizing its complicated feature extraction pipelines and enhancing explainability, benefiting future adaptation to more complex visual understanding tasks (e.g. video captioning). To achieve this task, we propose a novel decoupled two-stream framework for action recognition - HSAR, which utilizes high-semantic features for increased efficiency and provides well-founded explanations in terms of spatial-temporal perceptions that will benefit further expansions on visual understanding tasks. The inputs are decoupled into spatial and temporal streams with designated encoders aiming to extract only the pinnacle of representations, gaining high-semantic features while reducing computation costs greatly. A lightweight Temporal Motion Transformer (TMT) module is proposed for globally modeling temporal features through self-attention, omitting redundant spatial features. Decoupled spatial-temporal embeddings are further merged dynamically by an attention fusion model to form a joint high-semantic representation. The visualization of the attention in each module offers intuitive interpretations of HSAR’s explainability. Extensive experiments on three widely-used benchmarks (Kinetics400, 600, and Sthv2) show that our framework achieves high prediction accuracy with significantly reduced computation (only 64.07 GFLOPs per clip), offering a great trade-off between accuracy and computational costs.
CalorieCam360: Simultaneous Eating Action Recognition of Multiple People Using an Omnidirectional Camera

In recent years, as people become more health-conscious, dietary management has become increasingly important. Existing methods record only one person’s meals or eating movements, but cannot record the meals of multiple people at the same time. Therefore, we aim to record the meals of all people around a dining table using an omnidirectional camera simultaneously.
Intra-inter Modal Attention Blocks for RGB-D Semantic Segmentation

In this paper, we introduce a novel approach to address the challenge of effectively utilizing both RGB and depth information for semantic segmentation. Our approach, Intra-inter Modal Attention (IMA) blocks, considers both intra-modal and inter-modal aspects of the information to produce better results than prior methods which primarily focused on inter-modal relationships. The IMA blocks consist of a cross-modal non-local module and an adaptive channel-wise fusion module. The cross-modal non-local module captures both intra-modal and inter-modal variations at the spatial level through inter-modality parameter sharing, while the adaptive channel-wise fusion module refines the spatially-correlated features. Experimental results on RGB-D benchmark datasets demonstrate consistent performance improvements over various baseline segmentation networks when using the IMA blocks. Our in-depth analysis provides comprehensive results on the impact of intra-, inter-, and intra-inter modal attention on RGB-D segmentation.
MemeFier: Dual-stage Modality Fusion for Image Meme Classification

Hate speech is a societal problem that has significantly grown through the Internet. New forms of digital content such as image memes have given rise to spread of hate using multimodal means, being far more difficult to analyse and detect compared to the unimodal case. Accurate automatic processing, analysis and understanding of this kind of content will facilitate the endeavor of hindering hate speech proliferation through the digital world. To this end, we propose MemeFier, a deep learning-based architecture for fine-grained classification of Internet image memes, utilizing a dual-stage modality fusion module. The first fusion stage produces feature vectors containing modality alignment information that captures non-trivial connections between the text and image of a meme. The second fusion stage leverages the power of a Transformer encoder to learn inter-modality correlations at the token level and yield an informative representation. Additionally, we consider external knowledge as an additional input, and background image caption supervision as a regularizing component. Extensive experiments on three widely adopted benchmarks, i.e., Facebook Hateful Memes, Memotion7k and MultiOFF, indicate that our approach competes and in some cases surpasses state-of-the-art. Our code is available on GitHub1.
Multi-channel Convolutional Neural Network for Precise Meme Classification

This paper proposes a multi-channel convolutional neural network (MC-CNN) for classifying memes and non-memes. Our architecture is trained and validated on a challenging dataset that includes non-meme formats with textual attributes, which are also circulated online but rarely accounted for in meme classification tasks. Alongside a transfer learning base, two additional channels capture low-level and fundamental features of memes that make them unique from other images with text. We contribute an approach which outperforms previous meme classifiers specifically in live data evaluation, and one that is better able to generalise ‘in the wild’. Our research aims to improve accurate collation of meme content to support continued research in meme content analysis, and meme-related sub-tasks such as harmful content detection.
Dual-Path Semantic Construction Network for Composed Query-Based Image Retrieval

Composed Query-Based Image Retrieval (CQBIR) aims to retrieve the most relevant image from all the candidates according to the composed query. However, the multi-model query brings more challenges to learning the proper semantics, which include the traits mentioned in the text and resemblance with reference images. The improper learned semantics reduced the performance of existing CQBIR methods. To this end, we propose a novel framework termed Dual-Path Semantic Construction Network for Composed Query-Based Image Retrieval (DSCN). It consists of three components: (1) Multi-level Feature Extraction obtains the textual and visual features of various hierarchies for learning multi-level semantics. (2) Visual-to-Textual Semantic Construction module refines the learned semantics at the textual level. (3) Textual-to-Visual Semantic Construction module performs semantic guidance in the visual semantic space. Extensive experiments on three benchmarks, i.e., FashionIQ, Shoes, and Fashion200k show that our DSCN method outperforms recent state-of-the-art methods.
Knowledge-Aware Causal Inference Network for Visual Dialog

The effective knowledge and interaction within multi-modalities are key to Visual Dialog. Classic graph-based framework with the direct connection between history dialog and answer fails to give the right answer for the spurious guidance and strong bias induced from history dialog. Recent causal inference framework without this direct connection improves the generalization while worse accuracy. In this work, we propose a novel Knowledge-Aware Causal Inference framework(KACI-Net) in which the commonsense knowledge is introduced into the causal inference framework to achieve both high accuracy and generalization. Specifically, the commonsense knowledge is first generated according to the entities extracted from the question and fused with language and visual features with the co-attention to get the final answer. Comparisons with knowledge-unaware framework and graph-based knowledge-aware framework on VisDial v1.0 dataset show the superiority of our proposed framework and verify the effectiveness the usage of the commonsense knowledge for a good reasoning in Visual Dialog. Both high NDCG and MRR metrics indicate a good trade-off between accuracy and generalization.
CLAP: Contrastive Language-Audio Pre-training Model for Multi-modal Sentiment Analysis

Multi-modal Sentiment Analysis (MSA) is a hotspot of multi-modal fusion. To make full use of the correlation and complementarity between modalities in the process of fusing multi-modal data, we propose a two-stage framework of Contrastive Language-Audio Pre-training (CLAP) for the MSA task: 1) Making contrastive pre-training on an unlabeled large-scaled external data to yield better single-modal representations; 2) Adopting a Transformer-based multi-modal fusion module, to achieve further single-modal feature optimization and sentiment prediction via the task-driven training process. Our work fully demonstrates the importance and necessity of core elements such as pre-training, contrastive learning, and representation learning for the MSA task and significantly outperforms existing methods on two well-recognized MSA benchmarks.
Text-to-Image Fashion Retrieval with Fabric Textures

In this study, we proposed text-to-image fashion image retrieval that captures the texture of clothing fabrics. A fabric's texture is a major factor governing the comfort and appearance of clothes and significantly influences user preferences. However, unlike patterns and shapes that can readily be captured from a global image of the entire piece of clothing, extracting the fine and ambiguous characteristics of textures is considerably more challenging. The key concept is that by focusing on the "local" regions of clothing, detailed fabric textures can be more accurately captured. To this end, we propose a framework for learning cross-modal features from both global (the entire garment) and local (a close-up detail) image-text pairs. To verify the idea, we constructed a new dataset named Global and Local FACAD (G&L FACAD) by modifying the existing large-scale public FACAD dataset used for fashion retrieval. The experimental results confirm that the retrieval accuracy is significantly improved compared to the baselines. The code is available at https://github.com/SuzukiDaichi-git/texture_aware_fashion_retrieval.git.
A 6DoF VR Dataset of 3D virtualWorld for Privacy-Preserving Approach and Utility-Privacy Tradeoff

Virtual Reality (VR) applications offer an immersive user experience at the expense of privacy leakage caused by inevitably streaming various new types of user data. While some privacy-preserving approaches have been proposed for protecting one type of data, how to design and evaluate approaches for multiple types of user data are still open. On the other hand, preserving privacy will degrade the quality of experience of VR applications or say the utility of user data. How to achieve efficient utility-privacy tradeoff with multiple types of data is also open. Both call for a dataset that contains multiple types of user data and personal attributes of users as ground-truth values. In this paper, we collect a 6 degree-of-freedom VR dataset of 3D virtual worlds for the investigation of privacy-preserving approaches and utility-privacy tradeoff.
Fusion recommendation algorithm based on DeepWalk and Item-Based collaborative filtering

Collaborative filtering techniques have been applied to personalised recommendation systems in recommender systems. However, with the gradual increase in the number of platform users and goods, the user rating data on the entire project matrix is extremely sparse, resulting in the inaccuracy of the prediction results. DeepWalk, on the other hand, still performs well when the data is sparse. Therefore, DeepWalk is introduced into collaborative filtering, and a fusion recommendation algorithm based on DeepWalk and collaborative filtering is proposed to reduce the impact of cold start and sparsity on the recommendation results. Experiments show that the proposed fusion recommendation algorithm has higher recommendation accuracy when the data is sparse compared with other traditional algorithms, which improves the recommendation quality to a certain extent.
A Survey on Application of Knowledge Distillation in Healthcare Domain

Deep neural networks have become increasingly popular in both industry and academia due to the availability of vast amounts of data and computing resources. Transferlearning can be effective in cases where obtaining a large training dataset is not possible, allowing complex models trained on large datasets to be fine-tuned for specific tasks. However, the healthcare sector faces challenges due to the lack of readily available data and resources, and the complexity and memory requirements of pretrained models can make them difficult to deploy. To address these issues, knowledge distillation has been widely used in healthcare to compress large and complex models, making them easier to deploy. This study provides a comprehensive review of recent research that has utilized knowledge distillation to build deep learning models for various tasks in healthcare, as well as identifying potential areas for future research.
Holistic Knowledge Framework for Improving PC Design using Big Data Analytics

Cyberspace is massively expanding every day, and the users of these digital devices are looking for more innovative applications to ease their day-to-day work. The main objective of any device is to use available system resources at a maximum optimum level and give high performance to the users. Especially in personal computers (PC), there is still a massive gap in the user experience and utilization of available system resources. There are several situations where error conditions and system responsiveness lag. Over several years the digital world evolved and provided high-performance computing. Furthermore, the PC design enhancements could be carryout through the participation of PC users. That could be accomplished by collecting the system performance indicators, system behavior and properties, software application usage, user working log, and other wide range of real-time data from the PC. These telemetry data needed to be retrieved from an extensive array of PCs then these datasets would be reposited on the centralized servers. By applying the big data analytical methods and other mining algorithms over these extensive datasets to identify the hidden patterns and discover more knowledge about the key indicators that would help improve PC performance and user experiences. Through this knowledge acquisition model, the organizations and research institutions could develop the most resilient and adoptable PC designs and software applications developments.
A Comparative Study of Text Classification using Selective Machine Learning Algorithms

The process of categorizing documents into specific groups or categories is known as text classification (TC). This study focuses on implementing the Support Vector Machine (SVM) model for analyzing English text documents. Two separate experiments were conducted to test the efficacy of selected classifiers. The experimental results, based on a set of 1033 text documents, demonstrated that the SVM classifier achieved the best performance when the feature set size was small, while NB outperformed the other classifiers. Our findings showed that a classification rate of over 90% could be achieved by using 4000 more features.
Implementation of Cloud Computing in Higher Education: A Bibliographic Analysis

A network of remote servers is implemented by cloud computing, which is placed on the internet to safeguard, access, manage, and process data instead of on a local server. Cloud computing is currently a very in-demand service as it provides various benefits like huge computational power, cost cutting of services, enhanced performance, scalability, stability, and availability. In this review work 442 research papers on implementation/adoption of cloud computing in the field of higher education were explored using the bibliometric analysis and reviews of numerous studies were incorporated. These results demonstrate that during the past few years, there has been an increase in interest in cloud computing research, which has provided an improved understanding of the application or implementation or adaptation of cloud computing in the sphere of higher education. This report will give researchers, students, publishers, and experts information about the present cloud computing deployment trend in the higher education sector.
ChatBot using Python Flask

In order to obtain information about the college, such as the college fees, college facilities, the semester schedule, etc., students usually need to visit the college administration throughout the admissions process or as needed on a daily basis. In order to solve this issue, a chat bot may be created and built that can simply be connected with any college website in order to deliver relevant information about colleges. Chat-bots are software programs that conduct online chat conversations utilizing text or text-to-speech which is processed further, instead of putting you in direct contact with a real human agent. The user’s request is processed by a chat bot using natural language processing, which produces a thoughtful answer. The chat bot is also powered by an artificial intelligence system which itself decides the proper response to a certain query fired by the user based on the information contained in the database. The task of the college administration in giving information to students can be lessened by this system.
Improving Classification of Scanned Document Images using a Novel Combination of Pre-Processing Techniques

The accuracy of the optical character recognition (OCR) outputs is critical to the success of real-world applications. It is necessary to preprocess an image before an OCR system can correctly interpret the information contained therein. The technique presented employs a novel combination of preprocessing techniques to prepare document images for OCR. Text that has been extracted may be used in a range of ways, including business intelligence operations that often involve the extraction of meaningful semantic information from massive volumes of documents for subsequent strategic decision-making activities. This paper demonstrates a unique combination of image preprocessing on scanned documents which can be leveraged for semantic document understanding using LayoutLMv2. In the RVL-CDIP dataset's advertisement class, this study has included a few more samples. The proposed objective is to show that text classification systems can classify scanned documents accurately after applying proper preprocessing techniques. On the benchmark RVL-CDIP dataset, investigation indicates that the presented methodology outperforms the current state-of-the-art method by 2.34% of classification accuracy, achieving state-of-the-art results for document image classification.
Simulation and Performance Analysis of 3D Object Detection Algorithm using Deep Learning for Computer Vision Applications

3D object detection is becoming one of the main areas of research centered around Autonomous Vehicles. It involves real-time detection and tracking of 3D objects in a scenario. With the help of autonomous driving vehicles, one can ensure safety of human life and decrease the number of fatalities on road due to human error. Identified the problem with 3D object detection being that there is still a gap between 3D and 2D architecture. It is difficult to extract 3D object bounding boxes from monocular images because 2D images has lack of depth information, which is necessary for 3D object detection. In order to overcome this, a LiDAR approach is being used, which is based on the use of convolutional neural networks (CNNs) to get depth information. With one more dimension, the estimated depth map adds more information for 3D detection. Proposed work evaluated on standard benchmark datasets like KITTI dataset. 3D object detection systems are designed to provide 3D-oriented bounding boxes for 2D objects in 2D images. 8-corner, 3D centers with offsets, and 4-corner-2-height representations are used to parameterize 3D cuboids. Firstly, a 2D bounding box is created around the object estimated from the image. Then, the 3D LiDAR cloud points are superimposed on the image as an intermediate step to aid in 3D estimation. Using the cloud points, the depth map is formed and a 3D bounding box is created in x, y, and z plane. The proposed object detection model obtained an average accuracy of 98.1% on KITTI and custom dataset.
Newspaper Article Summarization Using Combinational Method

Text summarization produces a concise amount of data out of document which may represent the core information contained in the document. In this paper, a combinational approach has been used to generate summary from newspaper article that uses a number of features to rank sentences of the article. Graph based approach with minimal text sentiment analysis and the presence of category wise key phrases have been used. Sentences with the sentiment polarity of the overall document sentiment polarity can convey important information. Combination of multiple features can lead to the identification process of the significant sentences with which good summary can be formed. After the performance analysis of the approach, it has been observed that the technique can generate good summaries that are close to human written ones.
Bi-LSTM Neural Network Approach to Detect and Recognize Cyberthreats, Cyberstalking and Extremist Tweets in Twitter

Phishing attacks, in which victims are handed dangerous URLs, are among the cyberthreats. When you engage with these sites, a process of credential stealing begins. Furthermore, there has been an increase in the transmission of terrorist and extremist tweets, as well as cyberstalking operations, in recent days. As technology advances this can be addressed with machine learning approaches and artificial intelligence by developing models and conducting automated tweet identification. Cyberthreats, cyberstalking, and extremist comments are anticipated using this live algorithm. The dataset obtained from Kaggle is given as input to the model and are trained using the Bi-LSTM method based on a twitter dataset. The algorithm has outstanding performance scores, with a total accuracy of 93% and F1 score of 95%.