Miracle-dz/UniFAR-FTUs huggingface dataset:
Temporal Pyramid Transformer with Multimodal Interaction for Video Question Answering

Video question answering (VideoQA) is challenging given its multimodal combination of visual understanding and natural language understanding. While existing approaches seldom leverage the appearance-motion information in the video at multiple temporal scales, the interaction between the question and the visual information for textual semantics extraction is frequently ignored. Targeting these issues, this paper proposes a novel Temporal Pyramid Transformer (TPT) model with multimodal interaction for VideoQA. The TPT model comprises two modules, namely Question-specific Transformer (QT) and Visual Inference (VI). Given the temporal pyramid constructed from a video, QT builds the question semantics from the coarse-to-fine multimodal co-occurrence between each word and the visual content. Under the guidance of such question-specific semantics, VI infers the visual clues from the local-to-global multi-level interactions between the question and the video. Within each module, we introduce a multimodal attention mechanism to aid the extraction of question-video interactions, with residual connections adopted for the information passing across different levels. Through extensive experiments on three VideoQA datasets, we demonstrate better performances of the proposed method in comparison with the state-of-the-arts.
Asking It All: Generating Contextualized Questions for any Semantic Role

Asking questions about a situation is an inherent step towards understanding it. To this end, we introduce the task of role question generation, which, given a predicate mention and a passage, requires producing a set of questions asking about all possible semantic roles of the predicate. We develop a two-stage model for this task, which first produces a context-independent question prototype for each role and then revises it to be contextually appropriate for the passage. Unlike most existing approaches to question generation, our approach does not require conditioning on existing answers in the text. Instead, we condition on the type of information to inquire about, regardless of whether the answer appears explicitly in the text, could be inferred from it, or should be sought elsewhere. Our evaluation demonstrates that we generate diverse and well-formed questions for a large, broad-coverage ontology of predicates and roles.
Learning Hard Retrieval Decoder Attention for Transformers

The Transformer translation model is based on the multi-head attention mechanism, which can be parallelized easily. The multi-head attention network performs the scaled dot-product attention function in parallel, empowering the model by jointly attending to information from different representation subspaces at different positions. In this paper, we present an approach to learning a hard retrieval attention where an attention head only attends to one token in the sentence rather than all tokens. The matrix multiplication between attention probabilities and the value sequence in the standard scaled dot-product attention can thus be replaced by a simple and efficient retrieval operation. We show that our hard retrieval attention mechanism is 1.43 times faster in decoding, while preserving translation quality on a wide range of machine translation tasks when used in the decoder self- and cross-attention networks.
KNODE-MPC: A Knowledge-Based Data-Driven Predictive Control Framework for Aerial Robots

In this letter, we consider the problem of deriving and incorporating accurate dynamic models for model predictive control (MPC) with an application to quadrotor control. MPC relies on precise dynamic models to achieve the desired closed-loop performance. However, the presence of uncertainties in complex systems and the environments they operate in poses a challenge in obtaining sufficiently accurate representations of the system dynamics. In this letter, we make use of a deep learning tool, knowledge-based neural ordinary differential equations (KNODE), to augment a model obtained from first principles. The resulting hybrid model encompasses both a nominal first-principle model and a neural network learnt from simulated or real-world experimental data. Using a quadrotor, we benchmark our hybrid model against a state-of-the-art Gaussian Process (GP) model and show that the hybrid model provides more accurate predictions of the quadrotor dynamics and is able to generalize beyond the training data. To improve closed-loop performance, the hybrid model is integrated into a novel MPC framework, known as KNODE-MPC. Results show that the integrated framework achieves 60.2% improvement in simulations and more than 21% in physical experiments, in terms of trajectory tracking performance.
Directional MCLP Analysis and Reconstruction for Spatial Speech Communication

Spatial speech communication, i.e., the reconstruction of spoken signal along with the relative speaker position in the enclosure (reverberation information) is considered in this paper. Directional, diffuse components and the source position information are estimated at the transmitter, and perceptually effective reproduction is considered at the receiver. We consider spatially distributed microphone arrays for signal acquisition, and node specific signal estimation, along with its direction of arrival (DoA) estimation. Short-time Fourier transform (STFT) domain multi-channel linear prediction (MCLP) approach is used to model the diffuse component and relative acoustic transfer function is used to model the direct signal component. Distortion-less array response constraint and the time-varying complex Gaussian source model are used in the joint estimation of source DoA and the constituent signal components, separately at each node. The intersection between DoA directions at each node is used to compute the source position. Signal components computed at the node nearest to the estimated source position are taken as the signals for transmission. At the receiver, a four channel loud speaker (LS) setup is used for spatial reproduction, in which the source spatial image is reproduced relative to a chosen virtual listener position in the transmitter enclosure. Vector base amplitude panning (VBAP) method is used for direct component reproduction using the LS setup and the diffuse component is reproduced equally from all the loud speakers after decorrelation. This scheme of spatial speech communication is shown to be effective and more natural for hands-free telecommunication, through either loudspeaker listening or binaural headphone listening with head related transfer function (HRTF) based presentation.
Integrating Approaches to Word Representation

The problem of representing the atomic elements of language in modern neural learning systems is one of the central challenges of the field of natural language processing. I present a survey of the distributional, compositional, and relational approaches to addressing this task, and discuss various means of integrating them into systems, with special emphasis on the word level and the out-of-vocabulary phenomenon.
Euphemistic Phrase Detection by Masked Language Model

It is a well-known approach for fringe groups and organizations to use euphemisms -- ordinary-sounding and innocent-looking words with a secret meaning -- to conceal what they are discussing. For instance, drug dealers often use"pot"for marijuana and"avocado"for heroin. From a social media content moderation perspective, though recent advances in NLP have enabled the automatic detection of such single-word euphemisms, no existing work is capable of automatically detecting multi-word euphemisms, such as"blue dream"(marijuana) and"black tar"(heroin). Our paper tackles the problem of euphemistic phrase detection without human effort for the first time, as far as we are aware. We first perform phrase mining on a raw text corpus (e.g., social media posts) to extract quality phrases. Then, we utilize word embedding similarities to select a set of euphemistic phrase candidates. Finally, we rank those candidates by a masked language model -- SpanBERT. Compared to strong baselines, we report 20-50% higher detection accuracies using our algorithm for detecting euphemistic phrases.
AID: Efficient Prediction of Aggregated Intensity of Dependency in Large-scale Cloud Systems

Service reliability is one of the key challenges that cloud providers have to deal with. In cloud systems, unplanned service failures may cause severe cascading impacts on their dependent services, deteriorating customer satisfaction. Predicting the cascading impacts accurately and efficiently is critical to the operation and maintenance of cloud systems. Existing approaches identify whether one service depends on another via distributed tracing but no prior work focused on discriminating to what extent the dependency between cloud services is. In this paper, we survey the outages and the procedure for failure diagnosis in two cloud providers to motivate the definition of the intensity of dependency. We define the intensity of dependency between two services as how much the status of the callee service influences the caller service. Then we propose AID, the first approach to predict the intensity of dependencies between cloud services. AID first generates a set of candidate dependency pairs from the spans. AID then represents the status of each cloud service with a multivariate time series aggregated from the spans. With the representation of services, AID calculates the similarities between the statuses of the caller and the callee of each candidate pair. Finally, AID aggregates the similarities to produce a unified value as the intensity of the dependency. We evaluate AID on the data collected from an open-source microservice benchmark and a cloud system in production. The experimental results show that AID can efficiently and accurately predict the intensity of dependencies. We further demonstrate the usefulness of our method in a large-scale commercial cloud system.
Bodies Uncovered: Learning to Manipulate Real Blankets Around People via Physics Simulations

While robots present an opportunity to provide physical assistance to older adults and people with mobility impairments in bed, people frequently rest in bed with blankets that cover the majority of their body. To provide assistance for many daily self-care tasks, such as bathing, dressing, or ambulating, a caregiver must first uncover blankets from part of a person's body. In this work, we introduce a formulation for robotic bedding manipulation around people in which a robot uncovers a blanket from a target body part while ensuring the rest of the human body remains covered. We compare both reinforcement and supervised learning approaches for optimizing policies which provide a robot with grasp and release points that uncover a target part of the body. We trained and conducted evaluations of these policies in physics simulation environments that consist of a deformable cloth mesh covering a simulated human lying supine on a bed. In addition, we transfer simulation-trained policies to a real mobile manipulator and demonstrate that it can uncover a blanket from target body parts of a manikin lying in bed. Source code is available online.
LAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation

Pre-training visual and textual representations from large-scale image-text pairs is becoming a standard approach for many downstream vision-language tasks. The transformer-based models learn inter and intra-modal attention through a list of self-supervised learning tasks. This paper proposes LAViTeR, a novel architecture for visual and textual representation learning. The main module, Visual Textual Alignment (VTA) will be assisted by two auxiliary tasks, GAN-based image synthesis and Image Captioning. We also propose a new evaluation metric measuring the similarity between the learnt visual and textual embedding. The experimental results on two public datasets, CUB and MS-COCO, demonstrate superior visual and textual representation alignment in the joint feature embedding space
SYSTEMATIC TREATMENT OF PHOTO-NUCLEAR CROSS-SECTION CALCULATIONS

The photoneutron cross-sections for twelve radioactive elements are analyzed and are shown to follow an empirical formula. The aim of suggesting this mathematical formula is to be used in -ray incineration calculations, where the numerical effort is greatly reduced. The method used in the present research is simple and direct. The results of calculating incineration rate of selected radioactive isotopes were fitted to a simple polynomial form that takes into account the properties of the specified nucleus and the energy and flux intensity of the used -rays. Direct application of the present results on the selected radioactive isotopes shows that the suggested formula is efficient and accurate within accepted range of error. The radioactive elements selected represent a group of actinides resulting from nuclear fission reaction. A verity of -ray fluxes are selected that span a proper range of practical calculations. This leads to sixty fitted equations, with each equation containing five terms. This will provide enough accuracy and reliability for the suggested method. Fair comparisons with published work revealed that the current method gives satisfactory results.
Chemical abundances in Seyfert galaxies – VII. Direct abundance determination of neon based on optical and infrared emission lines

For the first time, neon abundance has been derived in the narrow line region from a sample of Seyfert 2 nuclei. In view of this, we compiled from the literature fluxes of optical and infrared (IR) narrow emission lines for 35 Seyfert 2 nuclei in the local universe ($z \:\lesssim \:0.06$). The relative intensities of emission lines were used to derive the ionic and total neon and oxygen abundances through electron temperature estimations (Te-method). For the neon, abundance estimates were obtained by using both Te-method and IR-method. Based on photoionization model results, we found a lower electron temperature [$t_{\rm e}({\rm Ne\, \small {III}})$] for the gas phase where the Ne2 + is located in comparison with t3 for the O2 + ion. We find that the differences (D) between Ne2 +/H+ ionic abundances calculated from IR-method and Te-method (assuming t3 in the Ne2 +/H+ derivation) are similar to the derivations in star-forming regions (SFs) and they are reduced by a mean factor of ∼3 when $t_{\rm e}({\rm Ne\, \small {III}})$ is considered. We propose a semi-empirical Ionization Correction Factor (ICF) for the neon, based on [Ne ii]12.81$\rm{\mu m}$, [Ne iii]15.56$\rm{\mu m}$, and oxygen ionic abundance ratios. We find that the average Ne/H abundance for the Seyfert 2s sample is nearly 2 times higher than similar estimate for SFs. Finally, for the very high metallicity regime (i.e. [$\rm 12+log(O/H)\: \gtrsim \: 8.80$]) an increase in Ne/O with O/H is found, which likely indicates secondary stellar production for the neon.
Kineverse: A Symbolic Articulation Model Framework for Model-Agnostic Mobile Manipulation

Service robots in the future need to execute abstract instructions such as “fetch the milk from the fridge”. To translate such instructions into actionable plans, robots require in-depth background knowledge. With regards to interactions with doors and drawers, robots require articulation models that they can use for state estimation and motion planning. Existing frameworks model articulated connections as abstract concepts such as prismatic, or revolute, but do not provide a parameterized model of these connections for computation. In this letter, we introduce a novel framework that uses symbolic mathematical expressions to model articulated structures – robots and objects alike – in a unified and extensible manner. We provide a theoretical description of this framework, and the operations that are supported by its models, and introduce an architecture to exchange our models in robotic applications, making them as flexible as any other environmental observation. To demonstrate the utility of our approach, we employ our practical implementation Kineverse for solving common robotics tasks from state estimation and mobile manipulation, and use it further in real-world mobile robot manipulation.
Visual Cues and Error Correction for Translation Robustness

Neural Machine Translation models are sensitive to noise in the input texts, such as misspelled words and ungrammatical construc-tions. Existing robustness techniques gener-ally fail when faced with unseen types of noise and their performance degrades on clean texts. In this paper, we focus on three types of realistic noise that are commonly generated by humans and introduce the idea of visual context to improve translation robustness for noisy texts. In addition, we describe a novel error correction training regime that can be used as an auxiliary task to further improve translation robustness. Experiments on English-French and English-German translation show that both multimodal and error correction com-ponents improve model robustness to noisy texts, while still retaining translation quality on clean texts.
FineAction: A Fine-Grained Video Dataset for Temporal Action Localization

Temporal action localization (TAL) is an important and challenging problem in video understanding. However, most existing TAL benchmarks are built upon the coarse granularity of action classes, which exhibits two major limitations in this task. First, coarse-level actions can make the localization models overfit in high-level context information, and ignore the atomic action details in the video. Second, the coarse action classes often lead to the ambiguous annotations of temporal boundaries, which are inappropriate for temporal action localization. To tackle these problems, we develop a novel large-scale and fine-grained video dataset, coined as FineAction, for temporal action localization. In total, FineAction contains 103K temporal instances of 106 action categories, annotated in 17K untrimmed videos. Compared to the existing TAL datasets, our FineAction takes distinct characteristics of fine action classes with rich diversity, dense annotations of multiple instances, and co-occurring actions of different classes, which introduces new opportunities and challenges for temporal action localization. To benchmark FineAction, we systematically investigate the performance of several popular temporal localization methods on it, and deeply analyze the influence of fine-grained instances in temporal action localization. As a minor contribution, we present a simple baseline approach for handling the fine-grained action detection, which achieves an mAP of 13.17% on our FineAction. We believe that FineAction can advance research of temporal action localization and beyond. The dataset is available at https://deeperaction.github. io/datasets/fineaction.
Phases of nonsupersymmetric gauge theories: The 
SO(Nc)
 case study

We investigate the IR phases of non-supersymmetric (non-SUSY) $SO(N_c)$ gauge theories with $N_F$ fermions in the vector representation obtained by perturbing the SUSY theory with anomaly mediated SUSY breaking (AMSB). We find that of the wide variety of phases appearing in the SUSY theory only two survive: for $N_F<\frac{3}{2} (N_c-2)$ the theory confines, breaking the $SU(N_F)$ global symmetry to $SO(N_F)$, while for $\frac{3}{2} (N_c-2)<N_F<3(N_c-2)$ the theory flows to a (super)-conformal fixed point. The abelian Coulomb and free magnetic phases do not survive and collapse to the confining phase. We also investigate the behavior of loop operators in order to provide a clear distinction between the confining and screened phases. With the choice of $Spin(N_c)$ for the global structure of the gauge group, we find that the electric Wilson loop indeed obeys an area law, providing one of the first demonstrations of true confinement with chiral symmetry breaking in a non-SUSY theory. We identify monopole condensation as the dynamics underlying confinement. These monopoles arise naturally for $N_F=N_c-2$. The case with smaller number of flavors can be obtained by integrating out flavors, and we confirm numerically that the monopole condensate persists in the presence of AMSB and mass perturbations.
On the shape of air–liquid interfaces with surface tension that bound rigidly rotating liquids in partially filled containers

The interface shape of a fluid in rigid body rotation about its axis and partially filling the container is often the subject of a homework problem in the first graduate fluids class. In that problem, surface tension is neglected, the interface shape is parabolic and the contact angle boundary condition is not satisfied in general. When surface tension is accounted for, the shapes exhibit much richer dependencies as a function of rotation velocity. We analyze steady interface shapes in rotating right-circular cylindrical containers under rigid body rotation in zero gravity. We pay especial attention to shapes near criticality, in which the interface, or part thereof, becomes straight and parallel to the axis of rotation at certain specific rotational speeds. We examine geometries where the container is axially infinite and derive properties of their solutions. We then examine in detail two special cases of menisci in a cylindrical container: a meniscus spanning the cross section; and a meniscus forming a bubble. In each case we develop exact solutions for the respective axial lengths as infinite series in powers of appropriate rotation parameters; and we find the respective asymptotic behaviors as the shapes approach their critical configuration. Finally we apply the method of asymptotic approximants to yield analytical expressions for the axial lengths of menisci over the whole range of rotation speeds. In this application, the analytical solution is employed to examine errors introduced by the assumption that the interface is a right circular cylinder; this assumption is key to the spinning bubble method used to measure surface tension.
NegatER: Unsupervised Discovery of Negatives in Commonsense Knowledge Bases

Codifying commonsense knowledge in machines is a longstanding goal of artificial intelligence. Recently, much progress toward this goal has been made with automatic knowledge base (KB) construction techniques. However, such techniques focus primarily on the acquisition of positive (true) KB statements, even though negative (false) statements are often also important for discriminative reasoning over commonsense KBs. As a first step toward the latter, this paper proposes NegatER, a framework that ranks potential negatives in commonsense KBs using a contextual language model (LM). Importantly, as most KBs do not contain negatives, NegatER relies only on the positive knowledge in the LM and does not require ground-truth negative examples. Experiments demonstrate that, compared to multiple contrastive data augmentation approaches, NegatER yields negatives that are more grammatical, coherent, and informative—leading to statistically significant accuracy improvements in a challenging KB completion task and confirming that the positive knowledge in LMs can be “re-purposed” to generate negative knowledge.