Reviewerly/exHarmony huggingface dataset:
Federated Learning for Personalized Image Aesthetics Assessment

Image aesthetics assessment (IAA) evaluates the generic aesthetic quality of images. Due to the subjectivity of IAA, personalized IAA (PIAA) is essential to offering dedicated image retrieval, editing, and recommendation services to individual users. However, existing PIAA approaches are trained under the centralized machine learning paradigm, which exposes sensitive image and rating data. To enhance PIAA in a privacy-preserving manner, we propose the first-of-its-kind Federated Learning-empowered Personalized Image Aesthetics Assessment (FedPIAA) approach with a simple yet effective model structure to capture image aesthetic patterns and personalized user aesthetic preferences. Extensive experimental comparison against eight baselines using the real-world dataset FLICKER-AES demonstrates that FedPIAA outperforms FedAvg by 1.56% under the small support set and by 4.86% under the large support set in terms of Spearman rank-order correlation coefficient between predicted and ground-truth personalized aesthetics scores, while achieving comparable performance with the best non-FL centralized PIAA approaches.
Difference-Aware Iterative Reasoning Network for Key Relation Detection

Scene graph serves as a crucial visual representation of an image, with salient objects providing richer semantics for detecting key relations. However, most methods use a one-step reasoning manner for key relation detection, which may not utilize potential clues effectively. Humans usually review and revise to achieve the final answer, and semantics of relations offer further linguistic clues. Therefore, we propose the Difference-aware Iterative Reasoning Network (DIRNet) to predict key relations in a multi-step manner. Our model estimates visual saliency, encodes contexts globally with message passing, and then refines predictions iteratively by considering the difference in predicted relation semantics and contextual information across iterations. Extensive experiments show that our model outperforms state-of-the-art methods in key relation prediction on the VG-KR benchmark, and achieves competitive results in common relation prediction on VG, demonstrating its generalization and superiority.
A Study of Model Based and Model Free Offline Reinforcement Learning

Reinforcement learning (RL) has received considerable attention for building autonomous systems with trained agents that interact with environments to learn optimal behavior. The RL requires a fundamentally online learning paradigm, one of the biggest obstacles for the widespread adoption of RL in scaling to many real-world scenarios. In this study we have explored the method of data collection and have applied offline model-free and model-based RL methods on classical OpenAIGym Cart-Pole environment. In this approach offline algorithms find a good policy from previously collected dataset.
Mask-Guided Stamp Erasure for Real Document Image

The application of text recognition in the automatic analysis of invoices, contracts and other documents has significantly raised office efficiency, but the stamps overlapping with the texts in these documents may seriously degrade the recognition accuracy. To mitigate the negative effect, we propose a stamp eraser, which can simultaneously remove the stamps and recover the occluded texts. To better distinguish stamps from the complex background, we propose a stamp localization module to generate fine-grained binary masks for the eraser to focus more on stamps. This module can also provide the background textual information for recovering text with a skip connection. We also propose the dilated mask to make the generated image look more natural by filling the stamp area with pixels around the stamp. In addition, to evaluate the effectiveness of our method in boosting text recognition, we propose a synthetic data set for training and a real dataset with complex background for testing. Experiments have shown that our method can effectively improve the recognition accuracy of the text occluded with the stamps.
Addressing Predicate Overlap in Scene Graph Generation with Semantic Granularity Controller

Semantic overlap between predicates (e.g., riding versus on) occurs inevitably when describing a scene. However, most existing Scene Graph Generation (SGG) works sidestep it by modeling the semantic overlap at category-level and assigning merely one-hot target to each sample, which hurt the performance on other reasonable predicates. In this paper, we argue that semantic overlap between predicates tends to vary in different abstract patterns, and a subject-object pair should retain multiple reasonable predicates. To this end, we make an early attempt to reformulate SGG as a partial multi-label learning problem and accordingly propose a model-agnostic Semantic Granularity Controller (SGC). SGC consists of a pattern-specific controller, partial multi-label learning, and controllable inference. The former two solve semantic confusion during training, while the latter makes the semantic granularity of prediction controllable. Extensive experiments demonstrate that SGC can improve the performance of SGG and guide the model to predict coarse/fine-grained predicates.
Towards Discriminative Semantic Relationship for Fine-grained Crowd Counting

As an extended task of crowd counting, fine-grained crowd counting aims to estimate the number of people in each semantic category instead of the whole in an image, and faces challenges including 1) inter-category crowd appearance similarity, 2) intra-category crowd appearance variations, and 3) frequent scene changes. In this paper, we propose a new fine-grained crowd counting approach named DSR to tackle these challenges by modeling Discriminative Semantic Relationship, which consists of two key components: Word Vector Module (WVM) and Adaptive Kernel Module (AKM). The WVM introduces more explicit semantic relationship information to better distinguish people of different semantic groups with similar appearance. The AKM dynamically adjusts kernel weights according to the features from different crowd appearance and scenes. The proposed DSR achieves superior results over state-of-the-art on the standard dataset. Our approach can serve as a new solid baseline and facilitate future research for the task of fine-grained crowd counting.
Sentence-Level Sign Language Recognition Framework

We present two solutions to sentence-level SLR. Sentence-level SLR requires mapping videos of sign language sentences to sequences of gloss labels. Connectionist Temporal Classification has been used as the classifier level of both models to avoid pre-segmenting the sentences into individual words. The first model is an LRCN-based model and the second model is a Multi-Cue Network. In the first approach, no prior knowledge has been leveraged. Raw frames are fed into an 18-layer LRCN with a CTC on top. In the second approach, three main characteristics (hand shape, hand position, and hand movement information) associated with each sign have been extracted using Mediapipe. 2D landmarks of hand shape have been used to create the skeleton of the hands and then are fed to a CONV-LSTM model. Hand locations and hand positions as relative distances to head are fed to separate LSTMs. All three sources of information have been then integrated into a Multi-Cue network with a CTC classification layer. We evaluate the performance of proposed models on RWTH-PHOENIX-Weather. After performing an excessive search on model hyper-parameters such as the number of feature maps, input size, batch size, sequence length, LSTM memory cell, regularization, and dropout, we achieve relatively low Word Error Rate.
Transfer Learning-Based Image Processing of Recognizing Office Objects for Indoor Mobile Robots

This paper proposes an OpenCV-based image processing method and a target recognition approach for indoor office mobile robots based on transfer learning. Firstly, the recognized image is pre-processed for clarity to remove the effect of uneven lighting; secondly, the transfer learning idea is introduced to extract image features, then the study modifies the fully connected layer of the Resnet18 network and trains the network model parameters; finally, the trained network model is used to conduct target recognition experiments on the pre-processed image. The experimental results show that the method can effectively recognize the images of items in indoor office mobile robots with relatively high accuracy and efficiency.
Dynamic Text Modeling and Categorization Framework based on Semantics Extraction and Similarity Checking

Semantics extraction is necessary to obtain relevant knowledge from general data. Recently, many researchers have developed numerous techniques to retrieve the semantics and use this information for categorization purposes. Primarily, the methodologies can be grouped into two main approaches: knowledge-based and corpus-based approaches. In this work, we studied how to achieve the same purpose using semantics extraction and matching methods. In order to prove the effectiveness of our approach, we utilized the problem of news article categorization as a use case. In fact, we developed a dynamic tool that adopts the proposed technique to crawl and classify articles from many well-known newspapers. Furthermore, the topic of articles can be grouped by their meta-information or by user-defined categories flawlessly. For generalization, the Universal Sentence Encoder model has been used for distributed semantic text representation and trained on a large processed Wikipedia corpus to achieve better accuracy. Besides, there is a possibility to enhance the results further using multi-label categorization or optimizing the meta-categories from the training dataset shortly.
Sparrow Tracer: Scalable Real Time Metrics from Event Log Pipelines at Twitter

Streaming event pipelines are one of the core components of Twitter's Data Infrastructure [1]. Twitter Sparrow is a project responsible for aggregating, processing and delivering user action generated events from microservices to data warehouses and data lakes in real time. User action generated events are converted into datasets used for data processing and data analytics use cases. This project is built using different on premise and cloud services. One of the important requirements of such a streaming event pipeline is the ability to measure important metrics such as latency, event count, event drop vs. success rate, and more. These metrics are responsible for defining the health of the streaming pipeline as well as providing valuable insights to users of the events. In this paper we introduce Sparrow Tracer which is a novel way to capture the event metrics using the concept of tracer events.
Anableps: Adapting Bitrate for Real-Time Communication Using VBR-encoded Video

Content providers increasingly replace traditional constant bitrate with variable bitrate (VBR) encoding in real-time video communication systems for better video quality. However, VBR encoding often leads to large and frequent bitrate fluctuation, inevitably deteriorating the efficiency of existing adaptive bitrate (ABR) methods. To tackle it, we propose the Anableps to consider the network dynamics and VBR-encoding-induced video bitrate fluctuations jointly for deploying the best ABR policy. With this aim, Anableps uses sender-side information from the past to predict the video bitrate range of upcoming frames. Such bitrate range is then combined with the receiver-side observations to set the proper bitrate target for video encoding using a reinforcement-learning-based ABR model. As revealed by extensive experiments on a real-world trace-driven testbed, our Anableps outperforms the de facto GCC with significant improvement of quality of experience, e.g., 1.88× video quality, 57% less bitrate consumption, 85% less stalling, and 74% shorter interaction delay.
Fully Automated Interpretable Breast Ultrasound Assisted Diagnosis System

Breast cancer is one of the most vulnerable malignant tumors for women in the world, which seriously threatens women's life and health. Fortunately, through timely screening, early breast cancer patients have a high cure rate, which is expected to reach complete recovery. Ultrasound imaging is a screening technology for breast cancer that is superior to the pathological judgment of puncture. It is less harmful to the patient and has the advantages of easy operation and real-time, so it is currently widely used. In order to alleviate the shortage of medical resources and improve screening efficiency, the technology of the breast cancer computer-aided diagnosis system came into being. This paper proposes a new and interpretable fully automated breast ultrasound computer-aided diagnosis method, which can automatically extract the corresponding medical features used by doctors after inputting an ultrasound image and infer the final diagnosis result through the knowledge association between these features. First, the input ultrasound image is read intelligently, similar to the reading process of ultrasound doctors. The diagnostic system extracts medical features useful for diagnosis from the ultrasound image. The second step is associating the extracted features with the known breast benign and malignant diagnosis knowledge graph and finally determining the benign and malignant tumor in this ultrasound image through the correlation relationship in the knowledge graph which is presented as a knowledge tensor. The experimental results show that the fully automated breast ultrasound CAD system proposed in this paper can realize the tradeoff between interpretability and accuracy.
Towards the Development of Text-Based Format Converters for Object Representation

Interoperability is the ability to communicate between applications that are written in different programming languages. The communication is established through the exchange of data, which has normally a representation in a standard format. The interoperability problem can be solved by implementing serializers from objects to standard formats, which can then be deserialized in any programming language. This paper presents a project that proposes a solution to address the interoperability problem, which is closely related to the implementation of object serializers. The project proposes to address two important aspects: 1) the representation of objects in standard text formats, independent of the programming language to allow interoperability between applications; and 2) the teaching of the processes involved to represent objects, through the incorporation of these topics in some of the courses of an undergraduate degree in Information Technologies and Systems.
ActionPrompt: Action-Guided 3D Human Pose Estimation With Text and Pose Prompting

Recent 2D-to-3D human pose estimation (HPE) utilizes temporal consistency across sequences to alleviate the depth ambiguity problem but ignore the action related prior knowledge hidden in the pose sequence. In this paper, we propose a plug-and-play module named Action Prompt Module (APM) that effectively mines different kinds of action clues for 3D HPE. The highlight is that, the mining scheme of APM can be widely adapted to different frameworks and bring consistent benefits. Specifically, we first present a novel Action-related Text Prompt module (ATP) that directly embeds action labels and transfers the rich language information in the label to the pose sequence. Besides, we further introduce Action-specific Pose Prompt module (APP) to mine the position-aware pose pattern of each action, and exploit the correlation between the mined patterns and input pose sequence for further pose refinement. Experiments show that APM can improve the performance of most video-based 2D-to-3D HPE frameworks by a large margin.
Image Template Matching via Dense and Consistent Contrastive Learning

Image template matching refers to localizing a small query image as opposed to a large reference image map. The query image a.k.a template has to be screened across every equal-sized region in the reference map to perform inner-product at pixel-level and the resulting similarity indicates the template location. Due to the domain heterogeneity between template and reference images, the matching performance degrades under dramatic appearance changes. More severely, the asymmetric matching easily leads to over-fitting by suggesting excessively false positive regions. To these ends, we propose an effective template matching method based on contrastive learning to perform a dense and consistent InfoNCEloss during matching. This can increase the matching at finer details, and thus effectively regularizes network training to prevent over-fitting. Extensive experiments on the synthetic aperture radar (SAR) and optical datasets, i.e., SEN1-2 and OS datasets demonstrate that our proposed method outperforms state-of-the-art methods by a large margin.
Moving RDBMS to NoSQL Paradigms

Current business requirements and application infrastructure paved the way toward not only SQL (NoSQL) paradigms. The business requirements impose on information technology that brings a new atmosphere for the apps that require handling massive amounts of heterogeneous data types in the limited time and provide Highly Available Transaction (HAT) requirements. Each of Relational Database Management System (RDBMS) and NoSQL models support a group of features that bring certain functionalities to utilize business requirements. The key features to determine a specific data model is technology capabilities to achieve business requirements. Due to the shortages of Atomicity, Consistency, Isolation, and Durability (ACID) transaction, relational databases are unable to manage the atmosphere's requirements, then, Brewer's theorem, data consistency, data availability and partition-tolerance (CAP) emerge an alternative to the modern business requirements. Traditional relational databases still support the applications that require to enforce relational transactions protocol, restrictions, and constraints. In parallel with both models, polyglot persistence has come out, coexisting with both models in the single web server or an enternrise server. This paper concentrates on the roles of various factors of moving toward NoSQL, such as handling big data, heterogeneous data types, Internet of Things (IoT), highly available transactions requirements, eschew ACID constraints, high performance, software-as-a-service business, dynamic modeling schema, and data scaling. Moreover, an overview of nature and modeling of both relational and non-relational databases, limitation of relational database, and reasoning to move to NoSQL is presented.
Cross-cycle Transformer-based Stitching Method for Low-resolution Borehole Images

The stitching of borehole images has an important predictive role in safety analysis in the field of geotechnical engineering and intelligent geological exploration. Applying traditional image stitching methods that designed specifically for high-resolution images to low-resolution images will lead to blurred stitching results, stitching seams, fewer matched feature points and difficulties in massive image stitching. To address these problems, we propose an autoencoder-based coarse-to-fine feature extraction network, which can extract image features with high semantic and improves the accuracy of the feature point matching. Besides, we design a cross-cycle Transformer-based image stitching framework, which increase the number of matching feature points by Cross-QuadTree attention and stitch image by affine transformation. Experimental results show that the proposed method can effectively stitch low-resolution geotechnical borehole images with satisfactory visual quality.
DFCP: Few-Shot DeepFake Detection via Contrastive Pretraining

Abuses of forgery techniques have created a considerable problem of misinformation on social media. Although scholars devote many efforts to face forgery detection (a.k.a DeepFake detection) and achieve some results, two issues still hinder the practical application. 1) Most detectors do not generalize well to unseen datasets. 2) In a supervised manner, most previous works require a considerable amount of manually labeled data. To address these problems, we propose a simple contrastive pertaining framework for DeepFake detection (DFCP), which works in a finetuning-after-pretraining manner, and requires only a few labels (5%). Specifically, we design a two-stream framework to simultaneously learn high-frequency texture features and high-level semantics information during pretraining. In addition, a video-based frame sampling strategy is proposed to mitigate potential noise data in the instance-discriminative contrastive learning to achieve better performance. Experimental results on several downstream datasets show the state-of-the-art performance of the proposed DFCP, which works at frame-level (w/o temporal reasoning) with high efficiency but outperforms video-level methods.
Body-Part Joint Detection and Association via Extended Object Representation

The detection of human body and its related parts (e.g., face, head or hands) have been intensively studied and greatly improved since the breakthrough of deep CNNs. However, most of these detectors are trained independently, making it a challenging task to associate detected body parts with people. This paper focuses on the problem of joint detection of human body and its corresponding parts. Specifically, we propose a novel extended object representation that integrates the center location offsets of body or its parts, and construct a dense single-stage anchor-based Body-Part Joint Detector (BPJDet). Body-part associations in BPJDet are embedded into the unified representation which contains both the semantic and geometric information. Therefore, BPJDet does not suffer from error-prone association post-matching, and has a better accuracy-speed trade-off. Furthermore, BPJDet can be seamlessly generalized to jointly detect any body part. To verify the effectiveness and superiority of our method, we conduct extensive experiments on the CityPersons, CrowdHuman and BodyHands datasets. The proposed BPJDet detector achieves state-of-the-art association performance on these three benchmarks while maintains high accuracy of detection. Code is released in https://github.com/hnuzhy/BPJDet.
EvenFace: Deep Face Recognition with Uniform Distribution of Identities

The development of loss functions over the past few years has brought great success to face recognition. Most algorithms focus on improving the intra-class compactness of face features but ignore the inter-class separability. In this paper, we propose a method named EvenFace, which introduces a regularization variance item and a mean term of inter-class separability to further promote the even distribution of class centers on the hypersphere, thereby increasing the inter-class distance. In order to evaluate the inter-class separability, a new index is proposed to better reflect the distribution of class centers and guide the classification. By penalizing the angle between each identity and its surrounding neighbors, the resulting uniform distribution of identities enables full exploitation of the feature space, leading to discriminative face representations. Our proposed loss function can effectively boost the performance of softmax loss variants. Quantitative comparisons with other state-of-the-art methods on several benchmarks demonstrate the superiority of EvenFace.