OpenRaiser/Intern-Atlas huggingface dataset:
Label-Attention Transformer with Geometrically Coherent Objects for Image Captioning
Venue: arXiv cs.CV 2021
Year: 2021

Automatic transcription of scene understanding in images and videos is a step towards artificial general intelligence. Image captioning is a nomenclature for describing meaningful information in an image using computer vision techniques. Automated image captioning techniques utilize encoder and decoder architecture, where the encoder extracts features from an image and the decoder generates a transcript. In this work, we investigate two unexplored ideas for image captioning using transformers: First, we demonstrate the enforcement of using objects' relevance in the surrounding environment. Second, learning an explicit association between labels and language constructs. We propose label-attention Transformer with geometrically coherent objects (LATGeO). The proposed technique acquires a proposal of geometrically coherent objects using a deep neural network (DNN) and generates captions by investigating their relationships using a label-attention module. Object coherence is defined using the localized ratio of the geometrical properties of the proposals. The label-attention module associates the extracted objects classes to the available dictionary using self-attention layers. The experimentation results show that objects' relevance in surroundings and binding of their visual feature with their geometrically localized ratios combined with its associated labels help in defining meaningful captions. The proposed framework is tested on the MSCOCO dataset, and a thorough evaluation resulting in overall better quantitative scores pronounces its superiority.
Frequent Itemset Mining with Multiple Minimum Supports: a Constraint-based Approach
Venue: arXiv cs.AI 2021
Year: 2021

The problem of discovering frequent itemsets including rare ones has received a great deal of attention. The mining process needs to be flexible enough to extract frequent and rare regularities at once. On the other hand, it has recently been shown that constraint programming is a flexible way to tackle data mining tasks. In this paper, we propose a constraint programming approach for mining itemsets with multiple minimum supports. Our approach provides the user with the possibility to express any kind of constraints on the minimum item supports. An experimental analysis shows the practical effectiveness of our approach compared to the state of the art.
Humanly Certifying Superhuman Classifiers
Venue: arXiv cs.LG 2021
Year: 2021

Estimating the performance of a machine learning system is a longstanding challenge in artificial intelligence research. Today, this challenge is especially relevant given the emergence of systems which appear to increasingly outperform human beings. In some cases, this "superhuman" performance is readily demonstrated; for example by defeating legendary human players in traditional two player games. On the other hand, it can be challenging to evaluate classification models that potentially surpass human performance. Indeed, human annotations are often treated as a ground truth, which implicitly assumes the superiority of the human over any models trained on human annotations. In reality, human annotators can make mistakes and be subjective. Evaluating the performance with respect to a genuine oracle may be more objective and reliable, even when querying the oracle is expensive or impossible. In this paper, we first raise the challenge of evaluating the performance of both humans and models with respect to an oracle which is unobserved. We develop a theory for estimating the accuracy compared to the oracle, using only imperfect human annotations for reference. Our analysis provides a simple recipe for detecting and certifying superhuman performance in this setting, which we believe will assist in understanding the stage of current research on classification. We validate the convergence of the bounds and the assumptions of our theory on carefully designed toy experiments with known oracles. Moreover, we demonstrate the utility of our theory by meta-analyzing large-scale natural language processing tasks, for which an oracle does not exist, and show that under our assumptions a number of models from recent years are with high probability superhuman.
A Quadratic Time Locally Optimal Algorithm for NP-hard Equal Cardinality Partition Optimization
Venue: arXiv cs.DS 2021
Year: 2021

We study the optimization version of the equal cardinality set partition problem (where the absolute difference between the equal sized partitions' sums are minimized). While this problem is NP-hard and requires exponential complexity to solve in general, we have formulated a weaker version of this NP-hard problem, where the goal is to find a locally optimal solution. The local optimality considered in our work is under any swap between the opposing partitions' element pairs. To this end, we designed an algorithm which can produce such a locally optimal solution in $O(N^2)$ time and $O(N)$ space. Our approach does not require positive or integer inputs and works equally well under arbitrary input precisions. Thus, it is widely applicable in different problem scenarios.
How Computer Science Can Aid Forest Restoration
Venue: arXiv cs.CY 2021
Year: 2021

The world faces two interlinked crises: climate change and loss of biodiversity. Forest restoration on degraded lands and surplus croplands can play a significant role both in sequestering carbon and re-establishing bio-diversity. There is a considerable body of research and practice that addresses forest restoration. However, there has been little work by computer scientists to bring powerful computational techniques to bear on this important area of work, perhaps due to a lack of awareness. In an attempt to bridge this gap, we present our vision of how techniques from computer science, broadly speaking, can aid current practice in forest restoration.
A Service for Supporting Digital and Immersive Cultural Experiences
Venue: arXiv cs.CY 2021
Year: 2021

Cultural heritage sites in Italy typically attract a large number of tourists every year. However, the lack of support for i) locating contents of interest; ii) discovering information on specific contents; and iii) ease of navigation within the heritage site; hinders the overall experience of the visitor. To this end, in this work, we present a Digital Object Space Management service developed as a part of the VASARI project. The service generates a digital twin (with 3D visualization) of a given cultural heritage site and further provides support for navigation and localization, thereby providing an immersive cultural experience to the visitor.
Ethics of AI: A Systematic Literature Review of Principles and Challenges
Venue: arXiv cs.CY 2021
Year: 2021

Ethics in AI becomes a global topic of interest for both policymakers and academic researchers. In the last few years, various research organizations, lawyers, think tankers and regulatory bodies get involved in developing AI ethics guidelines and principles. However, there is still debate about the implications of these principles. We conducted a systematic literature review (SLR) study to investigate the agreement on the significance of AI principles and identify the challenging factors that could negatively impact the adoption of AI ethics principles. The results reveal that the global convergence set consists of 22 ethical principles and 15 challenges. Transparency, privacy, accountability and fairness are identified as the most common AI ethics principles. Similarly, lack of ethical knowledge and vague principles are reported as the significant challenges for considering ethics in AI. The findings of this study are the preliminary inputs for proposing a maturity model that assess the ethical capabilities of AI systems and provide best practices for further improvements.
Device-to-System Performance Evaluation: from Transistor/Interconnect Modeling to VLSI Physical Design and Neural-Network Predictor
Venue: arXiv cs.ET 2021
Year: 2021

We present a DevIce-to-System Performance EvaLuation (DISPEL) workflow that integrates transistor and interconnect modeling, parasitic extraction, standard cell library characterization, logic synthesis, cell placement and routing, and timing analysis to evaluate system-level performance of new CMOS technologies. As the impact of parasitic resistances and capacitances continues to increase with dimensional downscaling, component-level optimization alone becomes insufficient, calling for a holistic assessment and optimization methodology across the boundaries between devices, interconnects, circuits, and systems. The physical implementation flow in DISPEL enables realistic analysis of complex wires and vias in VLSI systems and their impact on the chip power, speed, and area, which simple circuit simulations cannot capture. To demonstrate the use of DISPEL, a 32-bit commercial processor core is implemented using theoretical n-type MoS2 and p-type Black Phosphorous (BP) planar FETs at a projected 5-nm node, and the performance is benchmarked against Si FinFETs. While the superior gate control of the MoS2/BP FETs can theoretically provide 51% reduction in the iso-frequency energy consumption, the actual performance can be greatly limited by the source/drain contact resistances. With the large amount of data generated by DISPEL, a neural-network is trained to predict the key performance metrics of the 32-bit processor core using the characteristics of transistors and interconnects as the input features without the need to go through the time-consuming physical implementation flow. The machine learning algorithms show great potentials as a means for evaluation and optimization of new CMOS technologies and identifying the most significant technology design parameters.
Voter Perceptions of Trust in Risk-Limiting Audits
Venue: arXiv cs.CY 2021
Year: 2021

Risk-limiting audits (RLAs) are expected to strengthen the public confidence in the correctness of an election outcome. We hypothesize that this is not always the case, in part because for large margins between the winner and the runner-up, the number of ballots to be drawn can be so small that voters lose confidence. We conduct a user study with 105 participants resident in the US. Our findings confirm the hypothesis, showing that our study participants felt less confident when they were told the number of ballots audited for RLAs. We elaborate on our findings and propose recommendations for future use of RLAs.
Overview of Tencent Multi-modal Ads Video Understanding Challenge
Venue: arXiv cs.CV 2021
Year: 2021

Multi-modal Ads Video Understanding Challenge is the first grand challenge aiming to comprehensively understand ads videos. Our challenge includes two tasks: video structuring in the temporal dimension and multi-modal video classification. It asks the participants to accurately predict both the scene boundaries and the multi-label categories of each scene based on a fine-grained and ads-related category hierarchy. Therefore, our task has four distinguishing features from previous ones: ads domain, multi-modal information, temporal segmentation, and multi-label classification. It will advance the foundation of ads video understanding and have a significant impact on many ads applications like video recommendation. This paper presents an overview of our challenge, including the background of ads videos, an elaborate description of task and dataset, evaluation protocol, and our proposed baseline. By ablating the key components of our baseline, we would like to reveal the main challenges of this task and provide useful guidance for future research of this area. In this paper, we give an extended version of our challenge overview. The dataset will be publicly available at https://algo.qq.com/.
ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
Venue: arXiv cs.RO 2021
Year: 2021

Multisensory object-centric perception, reasoning, and interaction have been a key research topic in recent years. However, the progress in these directions is limited by the small set of objects available -- synthetic objects are not realistic enough and are mostly centered around geometry, while real object datasets such as YCB are often practically challenging and unstable to acquire due to international shipping, inventory, and financial cost. We present ObjectFolder, a dataset of 100 virtualized objects that addresses both challenges with two key innovations. First, ObjectFolder encodes the visual, auditory, and tactile sensory data for all objects, enabling a number of multisensory object recognition tasks, beyond existing datasets that focus purely on object geometry. Second, ObjectFolder employs a uniform, object-centric, and implicit representation for each object's visual textures, acoustic simulations, and tactile readings, making the dataset flexible to use and easy to share. We demonstrate the usefulness of our dataset as a testbed for multisensory perception and control by evaluating it on a variety of benchmark tasks, including instance recognition, cross-sensory retrieval, 3D reconstruction, and robotic grasping.
The NiuTrans System for WNGT 2020 Efficiency Task
Venue: arXiv cs.CL 2021
Year: 2021

This paper describes the submissions of the NiuTrans Team to the WNGT 2020 Efficiency Shared Task. We focus on the efficient implementation of deep Transformer models \cite{wang-etal-2019-learning, li-etal-2019-niutrans} using NiuTensor (https://github.com/NiuTrans/NiuTensor), a flexible toolkit for NLP tasks. We explored the combination of deep encoder and shallow decoder in Transformer models via model compression and knowledge distillation. The neural machine translation decoding also benefits from FP16 inference, attention caching, dynamic batching, and batch pruning. Our systems achieve promising results in both translation quality and efficiency, e.g., our fastest system can translate more than 40,000 tokens per second with an RTX 2080 Ti while maintaining 42.9 BLEU on \textit{newstest2018}. The code, models, and docker images are available at NiuTrans.NMT (https://github.com/NiuTrans/NiuTrans.NMT).
Locating Language-Specific Information in Contextualized Embeddings
Venue: arXiv cs.CL 2021
Year: 2021

Multilingual pretrained language models (MPLMs) exhibit multilinguality and are well suited for transfer across languages. Most MPLMs are trained in an unsupervised fashion and the relationship between their objective and multilinguality is unclear. More specifically, the question whether MPLM representations are language-agnostic or they simply interleave well with learned task prediction heads arises. In this work, we locate language-specific information in MPLMs and identify its dimensionality and the layers where this information occurs. We show that language-specific information is scattered across many dimensions, which can be projected into a linear subspace. Our study contributes to a better understanding of MPLM representations, going beyond treating them as unanalyzable blobs of information.
Generating Dataset For Large-scale 3D Facial Emotion Recognition
Venue: arXiv cs.CV 2021
Year: 2021

The tremendous development in deep learning has led facial expression recognition (FER) to receive much attention in the past few years. Although 3D FER has an inherent edge over its 2D counterpart, work on 2D images has dominated the field. The main reason for the slow development of 3D FER is the unavailability of large training and large test datasets. Recognition accuracies have already saturated on existing 3D emotion recognition datasets due to their small gallery sizes. Unlike 2D photographs, 3D facial scans are not easy to collect, causing a bottleneck in the development of deep 3D FER networks and datasets. In this work, we propose a method for generating a large dataset of 3D faces with labeled emotions. We also develop a deep convolutional neural network(CNN) for 3D FER trained on 624,000 3D facial scans. The test data comprises 208,000 3D facial scans.
FOMO: Topics versus documents in legal eDiscovery
Venue: arXiv cs.IR 2021
Year: 2021

In the United States, the parties to a lawsuit are required to search through their electronically stored information to find documents that are relevant to the specific case and produce them to their opposing party. Negotiations over the scope of these searches often reflect a fear that something will be missed (Fear of Missing Out: FOMO). A Recall level of 80%, for example, means that 20% of the relevant documents will be left unproduced. This paper makes the argument that eDiscovery is the process of identifying responsive information, not identifying documents. Documents are the carriers of the information; they are not the direct targets of the process. A given document may contain one or more topics or factoids and a factoid may appear in more than one document. The coupon collector's problem, Heaps law, and other analyses provide ways to model the problem of finding information from among documents. In eDiscovery, however, the parties do not know how many factoids there might be in a collection or their probabilities. This paper describes a simple model that estimates the confidence that a fact will be omitted from the produced set (the identified set), while being contained in the missed set. Two data sets are then analyzed, a small set involving microaggressions and larger set involving classification of web pages. Both show that it is possible to discover at least one example of each available topic within a relatively small number of documents, meaning the further effort will not return additional novel information. The smaller data set is also used to investigate whether the non-random order of searching for responsive documents commonly used in eDiscovery (called continuous active learning) affects the distribution of topics-it does not.
Field Study in Deploying Restless Multi-Armed Bandits: Assisting Non-Profits in Improving Maternal and Child Health
Venue: arXiv cs.LG 2021
Year: 2021

The widespread availability of cell phones has enabled non-profits to deliver critical health information to their beneficiaries in a timely manner. This paper describes our work to assist non-profits that employ automated messaging programs to deliver timely preventive care information to beneficiaries (new and expecting mothers) during pregnancy and after delivery. Unfortunately, a key challenge in such information delivery programs is that a significant fraction of beneficiaries drop out of the program. Yet, non-profits often have limited health-worker resources (time) to place crucial service calls for live interaction with beneficiaries to prevent such engagement drops. To assist non-profits in optimizing this limited resource, we developed a Restless Multi-Armed Bandits (RMABs) system. One key technical contribution in this system is a novel clustering method of offline historical data to infer unknown RMAB parameters. Our second major contribution is evaluation of our RMAB system in collaboration with an NGO, via a real-world service quality improvement study. The study compared strategies for optimizing service calls to 23003 participants over a period of 7 weeks to reduce engagement drops. We show that the RMAB group provides statistically significant improvement over other comparison groups, reducing ~ 30% engagement drops. To the best of our knowledge, this is the first study demonstrating the utility of RMABs in real world public health settings. We are transitioning our RMAB system to the NGO for real-world use.
Online Monitoring of Spatio-Temporal Properties for Imprecise Signals
Venue: arXiv cs.LO 2021
Year: 2021

From biological systems to cyber-physical systems, monitoring the behavior of such dynamical systems often requires to reason about complex spatio-temporal properties of physical and/or computational entities that are dynamically interconnected and arranged in a particular spatial configuration. Spatio-Temporal Reach and Escape Logic (STREL) is a recent logic-based formal language designed to specify and to reason about spatio-temporal properties. STREL considers each system's entity as a node of a dynamic weighted graph representing their spatial arrangement. Each node generates a set of mixed-analog signals describing the evolution over time of computational and physical quantities characterising the node's behavior. While there are offline algorithms available for monitoring STREL specifications over logged simulation traces, here we investigate for the first time an online algorithm enabling the runtime verification during the system's execution or simulation. Our approach extends the original framework by considering imprecise signals and by enhancing the logics' semantics with the possibility to express partial guarantees about the conformance of the system's behavior with its specification. Finally, we demonstrate our approach in a real-world environmental monitoring case study.
PrivateFetch: Scalable Catalog Delivery in Privacy-Preserving Advertising
Venue: arXiv cs.CR 2021
Year: 2021

In order to preserve the possibility of an Internet that is free at the point of use, attention is turning to new solutions that would allow targeted advertisement delivery based on behavioral information such as user preferences, without compromising user privacy. Recently, explorations in devising such systems either take approaches that rely on semantic guarantees like $k$-anonymity -- which can be easily subverted when combining with alternative information, and do not take into account the possibility that even knowledge of such clusters is privacy-invasive in themselves. Other approaches provide full privacy by moving all data and processing logic to clients -- but which is prohibitively expensive for both clients and servers. In this work, we devise a new framework called PrivateFetch for building practical ad-delivery pipelines that rely on cryptographic hardness and best-case privacy, rather than syntactic privacy guarantees or reliance on real-world anonymization tools. PrivateFetch utilizes local computation of preferences followed by high-performance single-server private information retrieval (PIR) to ensure that clients can pre-fetch ad content from servers, without revealing any of their inherent characteristics to the content provider. When considering an database of $>1,000,000$ ads, we show that we can deliver $30$ ads to a client in 40 seconds, with total communication costs of 192KB. We also demonstrate the feasibility of PrivateFetch by showing that the monetary cost of running it is less than 1% of average ad revenue. As such, our system is capable of pre-fetching ads for clients based on behavioral and contextual user information, before displaying them during a typical browsing session. In addition, while we test PrivateFetch as a private ad-delivery, the generality of our approach means that it could also be used for other content types.