Reviewerly/exHarmony huggingface dataset:
Evolution of AVIF Encoder: Speed and Memory Optimizations

The A VI Image File Format (A VIF) is an open-source, royalty-free image compression standard that is aimed at providing higher compression gains. In recent years, there has been an exponential increase in the amount of image data captured, stored, rendered and transmitted by billions of users across the globe, via several web and mobile applications. It is important to make image encoders faster and memory-efficient to cater to a wide range of devices ensuring quality user experience. In this paper, we provide an overview of speed optimizations that were contributed to the libaom encoder. We describe methods to reduce the complexity of prediction mode and residual-transform search. Experiments indicate that on average, the slowest libavif encoder speed setting provides a BD-Rate (PSNR) gain of 39%, whereas the fastest libavif encoder speed setting provides a BD-Rate (PSNR) gain of 16% with a 48% speed-up over the best quality WebP encoder for still images.
Development of an Image-Based Methodology for the Evaluation of Histopathological Features in Human Meningioma

Meningioma is the most common intracranial tumor in adulthood. With a clear female predominance and a recurrence rate that reaches 20%, it is, despite being considered a benign tumor, a pathology that greatly compromises post-diagnosis quality of life. Its prone to recur or progress to a higher degree is difficult to predict in the absence of obvious histological criteria. This project aims to develop an automatic methodology to aid in the diagnosis of meningiomas that is objective and easily reproducible. The methodology is based on histopathological image analysis using artificial intelligence and machine learning algorithms. It includes a semi-automatic process of identification and cleaning of the scanned samples, an automatic detection of the nuclei of each image and, finally, the parameterization of the samples. The obtained data together with the clinical information will be analyzed using statistical methods in order to provide a methodology to support clinical diagnosis and decision-making in patient management. The result is the development of an effective methodology that generates a set of data associated with morphological parameters with different trends according to the pathological groups studied. A tool has been developed that allows an effective semiautomatic analysis of the images to evaluate these parameters in an objective and reproducible way, helping in clinical decision-making and facilitating to undertake projects with large sample series. Clinical Relevance- The main contribution of this project is in the field of neuropathology, for the diagnosis of meningiomas, the most common brain tumor. The present project provides an objective and quantifiable prognosis methodology for the meningiomas, offering a more precise monitoring of the treatment applied to the patient, resulting in a better prognosis and better quality of life.
CRNet: Collaborative Refinement Network for Self-Supervised Video Object Segmentation

Most self-supervised based methods just rely on a point-to-point correspondence strategy to propagate masks through a video sequence. However, the pixel level matching is not sufficient and often results in noise. To ease the problem, we propose our collaborative refinement network (CRNet) for self-supervised video object segmentation. Our collaborative refinement network consists of two modules, i.e., memory retrieval module and collaborative refinement module. The memory retrieval module is used to perform point-to-point correspondence and produce a propagated mask for a query frame. The collaborative refinement module is designed to aggregate the reference & query information and learn the collaborative relationship among them implicitly to refine the output of the memory retrieval module. The whole model is trained from unlabeled video data without any human annotation in a self-supervised manner. Extensive experiments conducted on DAVIS-17 and YouTube-VOS demonstrate that our CRNet surpasses the state-of-the-art self-supervised methods and narrows the gap with the fully supervised methods.
Incremental Computation of Information Gain in Temporal Relational Streams

This paper studies a highly efficient statistical method in identifying anomalies and outliers in relational data streams. We compute information gain as a statistical measure of relational values in order to determine their significance. In contrast to more sophisticated model based methods, our algorithm can maintain the information gain measures incrementally in <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{O} (1)$</tex> cost, thus allowing an online monitoring of temporal relational streams.
Automated Adaptive Playback for Encoder-Adjudicated Live Sports

In low-latency live streaming, it is not easy to simultaneously avoid stalls and maintain low latency. The trade-off needs to be decided in real time. One of the best tools for controlling the buffer level (hence, the stalls) and live latency is adaptive playback speed control, which slows down the playback when the buffer level is critically low to reduce the risk of stalling and speeds up the playback when the buffer is sufficiently full to reduce the live latency. Adaptive playback generally works well. However, it may introduce artifacts depending on the content. Viewers may be more sensitive to playback speed changes at certain scenes. In this paper, we augment the state-of-the-art content-aware playback speed control (CAPSC) algorithm by developing a post-encoder add-on tool that analyzes the encoded live content and automatically tags each output chunk with a value indicating its suitability for playback speed changes. The source code for the CAPSC algorithm and the new post-encoder add-on (developed for the snooker games) is publicly available.
Explainability Audit: An Automated Evaluation of Local Explainability in Rooftop Image Classification

Explainable Artificial Intelligence (XAI) is a key concept in building trustworthy machine learning models. Local explainability methods seek to provide explanations for individual predictions. Usually, humans must check these explanations manually. When large numbers of predictions are being made, this approach does not scale. We address this deficiency for a rooftop classification problem specifically with ExplainabilityAu-dit, a method that automatically evaluates explanations generated by a local explainability toolkit and identifies rooftop images that require further auditing by a human expert. The proposed method utilizes explanations generated by the Local Interpretable Model-Agnostic Explanations (LIME) framework as the most important superpixels of each validation rooftop image during the prediction. Then a bag of image patches is extracted from the superpixels to determine their texture and evaluate the local explanations. Our results show that 95.7% of the cases to be audited are detected by the proposed system.
Fine-tuning the Robust Temporal Feature Magnitude Model for Enhancing the Accuracy of Anomaly Detection

Surveillance videos can capture a variety of realistic events and also anomalies. Due to an increase in the crime rate in public areas, surveillance cameras are adopted in a very large number. But as these crimes/public disputes are rare to occur at a specific location, human monitors are idle most of the time. Hence, there is a justified need to develop intelligent systems for anomaly detection. There are several seminal deep-neural architectures proposed in this field of anomaly detection ranging from using deep learning as a feature extraction tool to complete end-to-end deep-learning-based anomaly detection models. Any practical anomaly detection model must be generic in detecting a spectrum of anomalous events; however, several models can detect only specific types of anomalies. Further, several models are not amenable to distributed training over many machines on large streaming data, which is typical in a video surveillance system. In this paper, we discuss the techniques to detect anomalies in real-time by exploring recent architectures in the literature and analyze and explore ways we can improve the detection accuracy of the model. We propose a batching methodology that improves the existing model's area under the curve by 2%.
A News Recommendation Model Based on Time Awareness and News Relevance

Personalized news recommendation can target user interests and effectively alleviate information overload. Most of the existing methods are based on news content for recommendation, which mostly ignore the rich auxiliary information and neighbor information existing in real news recommendation scenarios. In addition, few methods provide easy-to-understand explanations. In this paper, we propose a news recommendation model based on time awareness and news relevance. The model combines various news auxiliary information and user-news interaction data in the form of heterogeneous graph, and mines the temporal relationship in the user click sequence for news recommendation. In addition, our model provide understandable recommendation explanations based on the multiple explanation bases extracted from the heterogeneous graph. Extensive experiments on two public and widely used datasets, Adressa and Globo, demonstrate both the effectiveness of the proposed approach and the reasonableness of recommendation explanations.
Evaluation Tool to Diagnose Faults and Discrepancy in Semi-Automated or Manual Annotations in Ultrasound Cine Loops (Videos)

Good quality (annotated) data is one of the most important aspects of supervised deep learning. Tasks such as semantic segmentation have a huge data requirement in exchange for only satisfactory performance. Large-scale annotations spread across multiple annotators tends to create inconsistencies, as there are various manual and semi-automated techniques involved. This mandates an external evaluator or expert to check and narrow down the problematic annotations. Studies have shown that even marking a few instances wrong in classification can lead to a significant performance drop in the model (mislabeling only 10% of one class can degrade the total performance of all classes by up to 10%). It has been noticed that fault localization by a medical expert is one of the most expensive and time-consuming processes. In this paper, we propose a novel framework for detecting the inconsistencies in the annotation of every object/anatomy in a specific image. We leverage the power of semi-supervised deep learning models (STCN) to help produce high-quality data for AI segmentation algorithms. Evaluation using this algorithm has been shown to reduce annotation review time by at least 5 hours for just 1000 images, and the quality of ground truth data improved thereby increasing the performance of the model by almost 3%.
Automatic labeling of the elements of a vulnerability report CVE with NLP

Common Vulnerabilities and Exposures (CVE) databases contain information about vulnerabilities of software products and source code. If individual elements of CVE descriptions can be extracted and structured, then the data can be used to search and analyze CVE descriptions. Herein we propose a method to label each element in CVE descriptions by applying Named Entity Recognition (NER). For NER, we used BERT, a transformer-based natural language processing model. Using NER with machine learning can label information from CVE descriptions even if there are some distortions in the data. An experiment involving manually prepared label information for 1000 CVE descriptions shows that the labeling accuracy of the proposed method is about 0.81 for precision and about 0.89 for recall. In addition, we devise a way to train the data by dividing it into labels. Our proposed method can be used to label each element automatically from CVE descriptions.
Text-Based Machine Learning Models for Cross-Domain Vulnerability Prediction: Why They may not be Effective?

Machine learning models are being utilized as applicable tools for source code vulnerability detection. However, the usage of models and techniques still vary in usability and prediction. In generalizing models to a more natural language approach, researchers have opted to train models on source code to identify existing and potential vulnerabilities. Exploratory research has been performed by treating source code as plain text. Additionally, models have been trained and tested within and across domains, such as projects. Based on a recent empirical study, text-based models are not effective across-domains. In this study, we examine results from this text-based approach to machine learning models. By providing a statistical analysis of data used and prediction results provided, we explore why the approach may not be sufficient for vulnerability prediction across domains.
An Automatic Evaluation Method for Open-domain Dialogue Based on BLEURT

Automatic open-domain dialogue generation is an important topic in natural language generation research. Due to the lack of good automatic evaluation methods, it is usually evaluated manually, which makes it difficult to compare different generation models. Automatic evaluation methods of natural language used in the past often required a reference corpus. However, for open-domain dialogue generation, the reference corpus will limit the possibilities of generation. In order to evaluate the quality of a generative model stably, we study the learning to evaluation approach to the generative dialogue based on deep learning method. Experimental corpus includes conversations collected from the web and conversations generated using the GPT-2 model. We manually evaluate these dialogue as a standard, and our system uses the BLEURT-20 model to learn an automatic evaluation model. We find t he model is suitable a san automatic evaluation mechanism for dialogue generation.
Healthcare Provider Summary Data for Fraud Classification

Fraud, waste, and abuse are spreading throughout the healthcare industry and costing patients and taxpayers billions of dollars. Fortunately, electronic medical records and publicly available data sources like the Centers for Medicare & Medicaid Services (CMS) have enabled data mining and machine learning techniques that can help automate the detection of healthcare fraud. In this study, we explore the application of healthcare provider summary data for the purpose of fraud detection. We leverage the latest CMS Part B Summary by Provider big data sets to curate two new labeled data sets for supervised learning. The two new data sets are compared to a popular baseline data set from related works using six runs of cross validation with two popular ensemble learners, multiple complementary performance metrics, and statistical tests. Classification results show that the proposed provider summary features are good indicators of healthcare fraud. A two-way analysis of variance test and 95% confidence intervals show that the new features yield significantly better performance on the fraud detection task when used to enrich existing data sets. Finally, feature contributions are measured with Shapley values to illustrate the top 20 features that contribute to fraud estimation.
Ultra-Sparse 360-Degree Camera View Synthesis for Immersive Virtual Tourism

360-degree video based virtual tours are becoming more and more popular due to travel costs and restrictions. Existing solutions leverage teleport, 3D modeling or image morphing, but none of them offers satisfactory immersion and scalability. In this paper, we propose a morphing based ultra-sparse 360-degree camera virtual tourism solution. It uses a novel bus tour mode to improve immersion; besides, it uses a series of strategies to improve feature matching such that morphing works well for ultra-sparse (15 m apart) cameras and the system can be deployed on a large scale. The experimental results show that our work results in remarkably better feature matching and synthesized views.
Towards Multimodal Semantic Consistency Analysis of Long Form Articles

With the rise of misinformation in news sites and social media, multi-modal machine learning methods that can identify fake news by analyzing inconsistencies within the articles have become increasingly important. Current state-of-the-art methods based on traditional image-caption models can only process captions within 1 to 2 sentences. Existing models struggle on analyzing long articles as they were not trained for such purposes. The main limitation is the lack of fine-grained localized evidence needed for consistency detection. We propose an ensemble method combining a bank of visual detectors and BERT-based NLP models that can effectively compute the consistency among the image(s) and paragraph(s) of texts. Our method is effective in both detecting the standard image-caption pairs and longer form news articles. Our method is able to process longer form of multi-modal media via the localization of fine-grained evidence with modularity and explainability. Evaluation is performed on a MS COCO data subset and a news article benchmark of the DARPA SemaFor program. We achieved 83% AUC on the COCO subset as well as a competitive result within the SemaFor evaluation.
CAPT: Contrastive Pre-Training based Semi-Supervised Open-Set Learning

Deep Semi-Supervised Learning (SSL) has shown very effective performance in recent years, such methods are typically under assumptions of a closed-world setting, where instances in the labeled and unlabeled data share the same class set. However, in real-world applications, samples with novel classes may be contained in the unlabeled data during the model deployment (open-set scenario), i.e. new types of scene images may occur in a self-driving system. In this paper, we advocate a two-stage semi-supervised learning approach CAPT, a framework for handling this realistic sce-nario based on a self-supervised pre-training step. The key idea is to introduce the embedding from pre-training into the SSL open-set classifier, so that the model can recognize the seen classes and cluster the instances from novel categories simultaneously. Our framework first pre-train a semantically meaningful representation of all samples from the labeled and unlabeled dataset. Next, CAPT applies the learned embedding as initialization to build a semisupervised classifier for clustering novel classes. We thoroughly evaluate our framework on large-scale image benchmarks CIFAR10, CIFAR100, obtaining state-of-the-art results.
Multi-Source Weak Supervision Fusion for Disaster Scene Recognition in Videos

Images or video recordings assist emergency responders in quickly inspecting the damage after a disaster event. New techniques are needed to help responders organize and find important information at the right time. However, most existing methods do not meet public safety standards due to a lack of training data. We propose a multi-source weak supervision fusion technique to train on a highly imbalanced dataset annotated with noisy labels. Using a Confident Learning technique, we reduce the noise effect while boosting the class labels' quality. We combine the predictive power from models trained on large-scale visual datasets using Differential Evolution. This research demonstrates a fully-automatic approach with great potential to reduce the required time and resources while delivering exceptional results. In the TRECVID2021 Disaster Scene Description and Indexing (DSDI) Challenge, our technique achieved the top score among all the submitted runs, independent of the training data utilized.
A Robust Collaborative Learning Framework Using Data Digests and Synonyms to Represent Absent Clients

We propose Collaborative Learning with Synonyms (CLSyn), a robust and versatile collaborative machine learning framework that can tolerate unexpected client absence during training while maintaining high model accuracy. Client absence during collaborative training can seriously degrade model performances, particularly for unbalanced and non-IID client data. We address this issue by introducing the notion of data digests of the training samples from the clients. The expansion of digests called synonyms can represent the original samples on the server and thus maintain overall model accuracy, even after the clients become unavailable. We compare our CLSyn implementations against three centralized Federated Learning algorithms, namely FedAvg, FedProx, and FedNova as baselines. Results on CIFAR-10, CIFAR-100, and EMNIST show that CLSyn consistently outperforms these baselines by significant margins in various client absence scenarios.
BAPO: A Large-Scale Multimodal Corpus for Ball Possession Prediction in American Football Games

We present a new task for multimodal information extraction: identify the players or teams that possess the ball in each play of an American football game. We also introduce BAPO, a large-scale corpus that consists of 100 games totaling around 200 hours of video broadcasts along with ball possession information for 15,132 plays. This corpus is rich and diverse because it involves a great number of players and different scenarios. BAPO poses a new challenge to build multimodal models that take into account language (what the broadcasters say), audio (broadcasters' tone, sounds from the audience, etc.) and vision (frames from the video broadcast). We further propose a baseline model, BAPOTer, and conduct comprehensive experiments. Results demonstrate that language is key to solve this task and that leveraging all three modalities is beneficial. The corpus is available at https://github.com/Isabella1118/BAPO.
Personalized Prediction of Indoor Comfort Using Graph Convolutional Matrix Completion

Recent progress in environment sensing technology focuses more on measuring the physical properties of the environment, e.g., temperature and noise, but lacks the ability to understand subjective responses, or feelings about the environment, e.g., indoor comfort. Feelings depend on both environmental conditions and individual needs and preferences. Different people may feel differently in the same room experiencing the same conditions. In this work, we apply a crowdsensing based approach to predict personalized indoor comfort. We assume that similar users share similar feelings about comfort, and that indoor comfort is related to a fixed set of conditions, e.g., space, humidity, temperature. We surveyed existing users of a case study building and used their responses to learn how to predict the personal responses of new users. Technically, we apply a graph convolutional matrix completion (GC-MC) method to predict the comfort of other users, by learning the dependency between the user profiles and their ratings to a fixed set of survey questions. We collect a kitchen survey dataset of 59 questions and in total 29 users of diverse profiles.
RE2L: A Real-World Dataset for Outdoor Low-Light Image Enhancement

Low- and normal light image pairs are usually required for training low-light image enhancement (LLIE) deep learning models. Although such training pairs can be synthesized by adjusting exposure, there is currently no outdoor real-world LLIE benchmark. Therefore, it is un-clear how to optimize various LLIE models for real out-door scenes. To fill the gap and benchmark outdoor LLIE tasks, we propose the REal-world two-dimensional Low-light dataset, dubbed RE2L. Natural-light illumination variations over both time and exposure level are fully captured in RE2L, which provides more information on illumination as restoration guidance during model training. We bench-marked the state-of-the-art LLIE methods over the proposed RE2L and its derivation RE2L-P datasets. Furthermore, we showed that using RE2L and RE2L-P for training LLIE models can improve their effectiveness both quantitatively and visually.