OpenRaiser/Intern-Atlas huggingface dataset:
Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning
Venue: arXiv cs.CV 2022
Year: 2022

Discriminativeness is a desirable feature of image captions: captions should describe the characteristic details of input images. However, recent high-performing captioning models, which are trained with reinforcement learning (RL), tend to generate overly generic captions despite their high performance in various other criteria. First, we investigate the cause of the unexpectedly low discriminativeness and show that RL has a deeply rooted side effect of limiting the output words to high-frequency words. The limited vocabulary is a severe bottleneck for discriminativeness as it is difficult for a model to describe the details beyond its vocabulary. Then, based on this identification of the bottleneck, we drastically recast discriminative image captioning as a much simpler task of encouraging low-frequency word generation. Hinted by long-tail classification and debiasing methods, we propose methods that easily switch off-the-shelf RL models to discriminativeness-aware models with only a single-epoch fine-tuning on the part of the parameters. Extensive experiments demonstrate that our methods significantly enhance the discriminativeness of off-the-shelf RL models and even outperform previous discriminativeness-aware methods with much smaller computational costs. Detailed analysis and human evaluation also verify that our methods boost the discriminativeness without sacrificing the overall quality of captions.
Learning the joint distribution of two sequences using little or no paired data
Venue: arXiv cs.LG 2022
Year: 2022

We present a noisy channel generative model of two sequences, for example text and speech, which enables uncovering the association between the two modalities when limited paired data is available. To address the intractability of the exact model under a realistic data setup, we propose a variational inference approximation. To train this variational model with categorical data, we propose a KL encoder loss approach which has connections to the wake-sleep algorithm. Identifying the joint or conditional distributions by only observing unpaired samples from the marginals is only possible under certain conditions in the data distribution and we discuss under what type of conditional independence assumptions that might be achieved, which guides the architecture designs. Experimental results show that even tiny amount of paired data (5 minutes) is sufficient to learn to relate the two modalities (graphemes and phonemes here) when a massive amount of unpaired data is available, paving the path to adopting this principled approach for all seq2seq models in low data resource regimes.
Counterfactual reasoning: Do language models need world knowledge for causal understanding?
Venue: arXiv cs.CL 2022
Year: 2022

Current pre-trained language models have enabled remarkable improvements in downstream tasks, but it remains difficult to distinguish effects of statistical correlation from more systematic logical reasoning grounded on understanding of the real world. In this paper we tease these factors apart by leveraging counterfactual conditionals, which force language models to predict unusual consequences based on hypothetical propositions. We introduce a set of tests drawn from psycholinguistic experiments, as well as larger-scale controlled datasets, to probe counterfactual predictions from a variety of popular pre-trained language models. We find that models are consistently able to override real-world knowledge in counterfactual scenarios, and that this effect is more robust in case of stronger baseline world knowledge -- however, we also find that for most models this effect appears largely to be driven by simple lexical cues. When we mitigate effects of both world knowledge and lexical cues to test knowledge of linguistic nuances of counterfactuals, we find that only GPT-3 shows sensitivity to these nuances, though this sensitivity is also non-trivially impacted by lexical associative factors.
Cheater's Bowl: Human vs. Computer Search Strategies for Open-Domain Question Answering
Venue: arXiv cs.CL 2022
Year: 2022

For humans and computers, the first step in answering an open-domain question is retrieving a set of relevant documents from a large corpus. However, the strategies that computers use fundamentally differ from those of humans. To better understand these differences, we design a gamified interface for data collection -- Cheater's Bowl -- where a human answers complex questions with access to both traditional and modern search tools. We collect a dataset of human search sessions, analyze human search strategies, and compare them to state-of-the-art multi-hop QA models. Humans query logically, apply dynamic search chains, and use world knowledge to boost searching. We demonstrate how human queries can improve the accuracy of existing systems and propose improving the future design of QA models.
Barriers to implementation of blockchain technology in agricultural supply chain
Venue: arXiv cs.CY 2022
Year: 2022

Emerging technologies, such as Blockchain and the Internet of Things (IoT), have had an immense role in propelling the agricultural industry towards the fourth agricultural revolution. Blockchain and IoT can greatly improve the traceability, efficiency, and safety of food along the supply chain. Given these contributions, there are many barriers to widespread adoption of this technology, including a deficit in many workers' ability to understand and effectively use this technology in addition to a lack of infrastructure to educate and support these workers. This paper discusses the barriers to adoption of blockchain and IoT technology in the agricultural supply chain. The authors analyse the impact of Blockchain and IoT in the food supply chain and methods in which governments and corporations can become more adaptable. Through the reduction in imports and protection of demand for local farmers, developing economies can create local sustainable agricultural ecosystems. Furthermore, the use of both public and private Research and Development can greatly contribute to the global knowledge on new technologies and improve many aspects of the food supply chain. In conclusion, both governments and corporations have a big role to play in the increased implementation of progressive technologies and the overall improvement of the food supply chain along with it.
E3C: A Tool for Evaluating Communication and Computation Costs in Authentication and Key Exchange Protocol
Venue: arXiv cs.CR 2022
Year: 2022

Today, with the development of blockchain and Internet of Things technologies, we need authentication protocols and key exchanges to communicate with these different technologies. Symmetric and asymmetric encryption methods are used to design authentication and key exchange protocols, each of which has different computation costs. In the Internet of Things systems, due to the limited memory and computation power, researchers are looking the lightweight design protocols so that the pressure caused by the computation of protocols can be minimized. Calculating protocols' computational and communication costs was done manually until now, which was associated with human error. In this paper, we proposed an E3C tool that can calculate the computation and communication costs of the authentication and key exchange protocols. E3C provides the ability to compare several protocols in terms of communication and processing costs and present them in separate charts. Comparing the processing and communication costs of classical and modern protocols manually and with the E3C indicate that the E3C can calculate the processing and communication costs of authentication and key exchange protocols with 99.99% accuracy.
A neural approach to synchronization in wireless networks with heterogeneous sources of noise
Venue: arXiv cs.NI 2022
Year: 2022

The paper addresses state estimation for clock synchronization in the presence of factors affecting the quality of synchronization. Examples are temperature variations and delay asymmetry. These working conditions make synchronization a challenging problem in many wireless environments, such as Wireless Sensor Networks or WiFi. Dynamic state estimation is investigated as it is essential to overcome non-stationary noises. The two-way timing message exchange synchronization protocol has been taken as a reference. No a-priori assumptions are made on the stochastic environments and no temperature measurement is executed. The algorithms are unequivocally specified offline, without the need of tuning some parameters in dependence of the working conditions. The presented approach reveals to be robust to a large set of temperature variations, different delay distributions and levels of asymmetry in the transmission path.
HADAS: Hardware-Aware Dynamic Neural Architecture Search for Edge Performance Scaling
Venue: arXiv cs.LG 2022
Year: 2022

Dynamic neural networks (DyNNs) have become viable techniques to enable intelligence on resource-constrained edge devices while maintaining computational efficiency. In many cases, the implementation of DyNNs can be sub-optimal due to its underlying backbone architecture being developed at the design stage independent of both: (i) the dynamic computing features, e.g. early exiting, and (ii) the resource efficiency features of the underlying hardware, e.g., dynamic voltage and frequency scaling (DVFS). Addressing this, we present HADAS, a novel Hardware-Aware Dynamic Neural Architecture Search framework that realizes DyNN architectures whose backbone, early exiting features, and DVFS settings have been jointly optimized to maximize performance and resource efficiency. Our experiments using the CIFAR-100 dataset and a diverse set of edge computing platforms have seen HADAS dynamic models achieve up to 57% energy efficiency gains compared to the conventional dynamic ones while maintaining the desired level of accuracy scores. Our code is available at https://github.com/HalimaBouzidi/HADAS
Contactless Oxygen Monitoring with Gated Transformer
Venue: arXiv cs.LG 2022
Year: 2022

With the increasing popularity of telehealth, it becomes critical to ensure that basic physiological signals can be monitored accurately at home, with minimal patient overhead. In this paper, we propose a contactless approach for monitoring patients' blood oxygen at home, simply by analyzing the radio signals in the room, without any wearable devices. We extract the patients' respiration from the radio signals that bounce off their bodies and devise a novel neural network that infers a patient's oxygen estimates from their breathing signal. Our model, called \emph{Gated BERT-UNet}, is designed to adapt to the patient's medical indices (e.g., gender, sleep stages). It has multiple predictive heads and selects the most suitable head via a gate controlled by the person's physiological indices. Extensive empirical results show that our model achieves high accuracy on both medical and radio datasets.
Last Mile of Blockchains: RPC and Node-as-a-service
Venue: arXiv cs.CR 2022
Year: 2022

While much research focuses on different methods to secure blockchain, information on the chain needs to be accessed by end-users to be useful. This position paper surveys different ways that end-users may access blockchains. We observe that between the two extremes of running a full node and fully utilizing a trusted third-party service, many solutions regarding light nodes are emerging. We analyze these solutions based on three basic properties of web communication: integrity, availability and privacy. We conclude that currently, the best way to access a blockchain while maintaining these three properties is still to run a full node. We consider it essential that future blockchain accessibility services should be built while considering these three expectations.
SAIH: A Scalable Evaluation Methodology for Understanding AI Performance Trend on HPC Systems
Venue: arXiv cs.DC 2022
Year: 2022

Novel artificial intelligence (AI) technology has expedited various scientific research, e.g., cosmology, physics and bioinformatics, inevitably becoming a significant category of workload on high performance computing (HPC) systems. Existing AI benchmarks tend to customize well-recognized AI applications, so as to evaluate the AI performance of HPC systems under predefined problem size, in terms of datasets and AI models. Due to lack of scalability on the problem size, static AI benchmarks might be under competent to help understand the performance trend of evolving AI applications on HPC systems, in particular, the scientific AI applications on large-scale systems. In this paper, we propose a scalable evaluation methodology (SAIH) for analyzing the AI performance trend of HPC systems with scaling the problem sizes of customized AI applications. To enable scalability, SAIH builds a set of novel mechanisms for augmenting problem sizes. As the data and model constantly scale, we can investigate the trend and range of AI performance on HPC systems, and further diagnose system bottlenecks. To verify our methodology, we augment a cosmological AI application to evaluate a real HPC system equipped with GPUs as a case study of SAIH.
"just like therapy!": Investigating the Potential of Storytelling in Online Postpartum Depression Communities
Venue: arXiv cs.HC 2022
Year: 2022

One in seven women encounter postpartum depression upon transitioning to motherhood. Many of them frequently seek social support in online support groups. We conducted a mixed-methods formative research to assess the potential of digital storytelling on those online platforms. We observed that mothers acquire social support from online groups through storytelling. We present design recommendations for online postpartum depression (PPD) communities to utilize storytelling in fostering emotional support and providing relevant information and education through storytelling.
Tree DNN: A Deep Container Network
Venue: arXiv cs.LG 2022
Year: 2022

Multi-Task Learning (MTL) has shown its importance at user products for fast training, data efficiency, reduced overfitting etc. MTL achieves it by sharing the network parameters and training a network for multiple tasks simultaneously. However, MTL does not provide the solution, if each task needs training from a different dataset. In order to solve the stated problem, we have proposed an architecture named TreeDNN along with it's training methodology. TreeDNN helps in training the model with multiple datasets simultaneously, where each branch of the tree may need a different training dataset. We have shown in the results that TreeDNN provides competitive performance with the advantage of reduced ROM requirement for parameter storage and increased responsiveness of the system by loading only specific branch at inference time.
Patterns of Sociotechnical Design Preferences of Chatbots for Intergenerational Collaborative Innovation : A Q Methodology Study
Venue: arXiv cs.HC 2022
Year: 2022

Chatbot technology is increasingly emerging as a virtual assistant. Chatbots could allow individuals and organizations to accomplish objectives that are currently not fully optimized for collaboration across an intergenerational context. This paper explores the preferences of chatbots as a companion in intergenerational innovation. The Q methodology was used to investigate different types of collaborators and determine how different choices occur between collaborators that merge the problem and solution domains of chatbots' design within intergenerational settings. The study's findings reveal that various chatbot design priorities are more diverse among younger adults than senior adults. Additionally, our research further outlines the principles of chatbot design and how chatbots will support both generations. This research is the first step towards cultivating a deeper understanding of different age groups' subjective design preferences for chatbots functioning as a companion in the workplace. Moreover, this study demonstrates how the Q methodology can guide technological development by shifting the approach from an age-focused design to a common goal-oriented design within a multigenerational context.
SimVTP: Simple Video Text Pre-training with Masked Autoencoders
Venue: arXiv cs.CV 2022
Year: 2022

This paper presents SimVTP: a Simple Video-Text Pretraining framework via masked autoencoders. We randomly mask out the spatial-temporal tubes of input video and the word tokens of input text and then feed them into a unified autencoder to reconstruct the missing pixels and words. Our SimVTP has several properties: 1) Thanks to the unified autoencoder, SimVTP reconstructs the masked signal of one modality with the help from another modality, which implicitly learns the cross-modal alignment between video tubes and text tokens. 2) SimVTP not only benefits from a high video masking ratio (e.g. 90%) due to the temporal redundancy of video, but also needs a high text masking ratio (e.g. 75%), which is much higher than BERT (e.g. 15%), to achieve optimal performance. This is because the aid of video modality makes text reconstruction less challenging, which thus needs a higher mask ratio to make the pretext harder for useful feature learning. 3) Equipping SimVTP with video-text contrastive learning (VTC) and video-text matching (VTM), which are two commonly used cross-modal training strategies, could further improve the transferable performance significantly. 4) SimVTP is dataefficent, e.g., pre-training only on 10% data of WebVid-2M, SimVTP achieves surprisingly good results (43.8 R@1) on MSRVTT, which is far above recent state-of-the-art methods pre-trained on both CC3M and WebVid-2M. We transfer our pre-trained model to various downstream tasks and achieve superior performance. The codes and models will be released at https://github.com/mayuelala/SimVTP.
Low-Resource End-to-end Sanskrit TTS using Tacotron2, WaveGlow and Transfer Learning
Venue: arXiv cs.CL 2022
Year: 2022

End-to-end text-to-speech (TTS) systems have been developed for European languages like English and Spanish with state-of-the-art speech quality, prosody, and naturalness. However, development of end-to-end TTS for Indian languages is lagging behind in terms of quality. The challenges involved in such a task are: 1) scarcity of quality training data; 2) low efficiency during training and inference; 3) slow convergence in the case of large vocabulary size. In our work reported in this paper, we have investigated the use of fine-tuning the English-pretrained Tacotron2 model with limited Sanskrit data to synthesize natural sounding speech in Sanskrit in low resource settings. Our experiments show encouraging results, achieving an overall MOS of 3.38 from 37 evaluators with good Sanskrit spoken knowledge. This is really a very good result, considering the fact that the speech data we have used is of duration 2.5 hours only.
Learning rigid dynamics with face interaction graph networks
Venue: arXiv cs.LG 2022
Year: 2022

Simulating rigid collisions among arbitrary shapes is notoriously difficult due to complex geometry and the strong non-linearity of the interactions. While graph neural network (GNN)-based models are effective at learning to simulate complex physical dynamics, such as fluids, cloth and articulated bodies, they have been less effective and efficient on rigid-body physics, except with very simple shapes. Existing methods that model collisions through the meshes' nodes are often inaccurate because they struggle when collisions occur on faces far from nodes. Alternative approaches that represent the geometry densely with many particles are prohibitively expensive for complex shapes. Here we introduce the Face Interaction Graph Network (FIGNet) which extends beyond GNN-based methods, and computes interactions between mesh faces, rather than nodes. Compared to learned node- and particle-based methods, FIGNet is around 4x more accurate in simulating complex shape interactions, while also 8x more computationally efficient on sparse, rigid meshes. Moreover, FIGNet can learn frictional dynamics directly from real-world data, and can be more accurate than analytical solvers given modest amounts of training data. FIGNet represents a key step forward in one of the few remaining physical domains which have seen little competition from learned simulators, and offers allied fields such as robotics, graphics and mechanical design a new tool for simulation and model-based planning.
LSVL: Large-scale season-invariant visual localization for UAVs
Venue: arXiv cs.RO 2022
Year: 2022

Localization of autonomous unmanned aerial vehicles (UAVs) relies heavily on Global Navigation Satellite Systems (GNSS), which are susceptible to interference. Especially in security applications, robust localization algorithms independent of GNSS are needed to provide dependable operations of autonomous UAVs also in interfered conditions. Typical non-GNSS visual localization approaches rely on known starting pose, work only on a small-sized map, or require known flight paths before a mission starts. We consider the problem of localization with no information on initial pose or planned flight path. We propose a solution for global visual localization on a map at scale up to 100 km2, based on matching orthoprojected UAV images to satellite imagery using learned season-invariant descriptors. We show that the method is able to determine heading, latitude and longitude of the UAV at 12.6-18.7 m lateral translation error in as few as 23.2-44.4 updates from an uninformed initialization, also in situations of significant seasonal appearance difference (winter-summer) between the UAV image and the map. We evaluate the characteristics of multiple neural network architectures for generating the descriptors, and likelihood estimation methods that are able to provide fast convergence and low localization error. We also evaluate the operation of the algorithm using real UAV data and evaluate running time on a real-time embedded platform. We believe this is the first work that is able to recover the pose of an UAV at this scale and rate of convergence, while allowing significant seasonal difference between camera observations and map.
On the Global Solution of Soft k-Means
Venue: arXiv cs.LG 2022
Year: 2022

This paper presents an algorithm to solve the Soft k-Means problem globally. Unlike Fuzzy c-Means, Soft k-Means (SkM) has a matrix factorization-type objective and has been shown to have a close relation with the popular probability decomposition-type clustering methods, e.g., Left Stochastic Clustering (LSC). Though some work has been done for solving the Soft k-Means problem, they usually use an alternating minimization scheme or the projected gradient descent method, which cannot guarantee global optimality since the non-convexity of SkM. In this paper, we present a sufficient condition for a feasible solution of Soft k-Means problem to be globally optimal and show the output of the proposed algorithm satisfies it. Moreover, for the Soft k-Means problem, we provide interesting discussions on stability, solutions non-uniqueness, and connection with LSC. Then, a new model, named Minimal Volume Soft k-Means (MVSkM), is proposed to address the solutions non-uniqueness issue. Finally, experimental results support our theoretical results.