Miracle-dz/UniFAR-FTUs huggingface dataset:
Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teachers

The ability to pick up on language signals in an ongoing interaction is crucial for future machine learning models to collaborate and interact with humans naturally. In this paper, we present an initial study that evaluates intra-episodic feedback given in a collaborative setting. We use a referential language game as a controllable example of a task-oriented collaborative joint activity. A teacher utters a referring expression generated by a well-known symbolic algorithm (the"Incremental Algorithm") as an initial instruction and then monitors the follower's actions to possibly intervene with intra-episodic feedback (which does not explicitly have to be requested). We frame this task as a reinforcement learning problem with sparse rewards and learn a follower policy for a heuristic teacher. Our results show that intra-episodic feedback allows the follower to generalize on aspects of scene complexity and performs better than providing only the initial statement.
Movie101: A New Movie Understanding Benchmark

To help the visually impaired enjoy movies, automatic movie narrating systems are expected to narrate accurate, coherent, and role-aware plots when there are no speaking lines of actors. Existing works benchmark this challenge as a normal video captioning task via some simplifications, such as removing role names and evaluating narrations with ngram-based metrics, which makes it difficult for automatic systems to meet the needs of real application scenarios. To narrow this gap, we construct a large-scale Chinese movie benchmark, named Movie101. Closer to real scenarios, the Movie Clip Narrating (MCN) task in our benchmark asks models to generate role-aware narration paragraphs for complete movie clips where no actors are speaking. External knowledge, such as role information and movie genres, is also provided for better movie understanding. Besides, we propose a new metric called Movie Narration Score (MNScore) for movie narrating evaluation, which achieves the best correlation with human evaluation. Our benchmark also supports the Temporal Narration Grounding (TNG) task to investigate clip localization given text descriptions. For both two tasks, our proposed methods well leverage external knowledge and outperform carefully designed baselines. The dataset and codes are released at https://github.com/yuezih/Movie101.
CrowdWeb: A Visualization Tool for Mobility Patterns in Smart Cities

Human mobility patterns refer to the regularities and trends in the way people move, travel, or navigate through different geographical locations over time. Detecting human mobility patterns is essential for a variety of applications, including smart cities, transportation management, and disaster response. The accuracy of current mobility prediction models is less than 25%. The low accuracy is mainly due to the fluid nature of human movement. Typically, humans do not adhere to rigid patterns in their daily activities, making it difficult to identify hidden regularities in their data. To address this issue, we proposed a web platform to visualize human mobility patterns by abstracting the locations into a set of places to detect more realistic patterns. However, the platform was initially designed to detect individual mobility patterns, making it unsuitable for representing the crowd in a smart city scale. Therefore, we extend the platform to visualize the mobility of multiple users from a city-scale perspective. Our platform allows users to visualize a graph of visited places based on their historical records using a modified PrefixSpan approach. Additionally, the platform synchronizes, aggregates, and displays crowd mobility patterns across various time intervals within a smart city. We showcase our platform using a real dataset.
Ownership Chains in Multinational Enterprises

In this contribution, we investigate the role of ownership chains developed by multinational enterprises across different national borders. First, we document that parent companies control a majority (58%) of foreign subsidiaries through indirect control relationships involving at least two countries along an ownership chain. Therefore, we hypothesize that locations along ownership chains are driven by the existence of communication costs to transmit management decisions. In line with motivating evidence, we develop a theoretical model for competition on corporate control that considers the possibility that parent companies in the origin countries can delegate their monitoring activities in final subsidiaries to middlemen subsidiaries that are located in intermediate jurisdictions. Our model returns us a two-step empirical strategy with two gravity equations: i) a triangular gravity for establishing a middleman by the parent, conditional on final investments' locations; ii) a classical gravity for the location of final investments. First estimates confirm the predictions that ease of communication at the country level shapes the heterogeneous locations of subsidiaries along global ownership chains.
Offline Reinforcement Learning with Additional Covering Distributions

We study learning optimal policies from a logged dataset, i.e., offline RL, with function approximation. Despite the efforts devoted, existing algorithms with theoretic finite-sample guarantees typically assume exploratory data coverage or strong realizable function classes, which is hard to be satisfied in reality. While there are recent works that successfully tackle these strong assumptions, they either require the gap assumptions that only could be satisfied by part of MDPs or use the behavior regularization that makes the optimality of learned policy even intractable. To solve this challenge, we provide finite-sample guarantees for a simple algorithm based on marginalized importance sampling (MIS), showing that sample-efficient offline RL for general MDPs is possible with only a partial coverage dataset and weak realizable function classes given additional side information of a covering distribution. Furthermore, we demonstrate that the covering distribution trades off prior knowledge of the optimal trajectories against the coverage requirement of the dataset, revealing the effect of this inductive bias in the learning processes.
Explaining How Transformers Use Context to Build Predictions

Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model’s prediction, it is still unclear how prior words affect the model’s decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions.
VAKTA-SETU: A Speech-to-Speech Machine Translation Service in Select Indic Languages

In this work, we present our deployment-ready Speech-to-Speech Machine Translation (SSMT) system for English-Hindi, English-Marathi, and Hindi-Marathi language pairs. We develop the SSMT system by cascading Automatic Speech Recognition (ASR), Disfluency Correction (DC), Machine Translation (MT), and Text-to-Speech Synthesis (TTS) models. We discuss the challenges faced during the research and development stage and the scalable deployment of the SSMT system as a publicly accessible web service. On the MT part of the pipeline too, we create a Text-to-Text Machine Translation (TTMT) service in all six translation directions involving English, Hindi, and Marathi. To mitigate data scarcity, we develop a LaBSE-based corpus filtering tool to select high-quality parallel sentences from a noisy pseudo-parallel corpus for training the TTMT system. All the data used for training the SSMT and TTMT systems and the best models are being made publicly available. Users of our system are (a) Govt. of India in the context of its new education policy (NEP), (b) tourists who criss-cross the multilingual landscape of India, (c) Indian Judiciary where a leading cause of the pendency of cases (to the order of 10 million as on date) is the translation of case papers, (d) farmers who need weather and price information and so on. We also share the feedback received from various stakeholders when our SSMT and TTMT systems were demonstrated in large public events.
Contrastive Language-Image Pretrained Models are Zero-Shot Human Scanpath Predictors

Understanding the mechanisms underlying human attention is a fundamental challenge for both vision science and artificial intelligence. While numerous computational models of free-viewing have been proposed, less is known about the mechanisms underlying task-driven image exploration. To address this gap, we present CapMIT1003, a database of captions and click-contingent image explorations collected during captioning tasks. CapMIT1003 is based on the same stimuli from the well-known MIT1003 benchmark, for which eye-tracking data under free-viewing conditions is available, which offers a promising opportunity to concurrently study human attention under both tasks. We make this dataset publicly available to facilitate future research in this field. In addition, we introduce NevaClip, a novel zero-shot method for predicting visual scanpaths that combines contrastive language-image pretrained (CLIP) models with biologically-inspired neural visual attention (NeVA) algorithms. NevaClip simulates human scanpaths by aligning the representation of the foveated visual stimulus and the representation of the associated caption, employing gradient-driven visual exploration to generate scanpaths. Our experimental results demonstrate that NevaClip outperforms existing unsupervised computational models of human visual attention in terms of scanpath plausibility, for both captioning and free-viewing tasks. Furthermore, we show that conditioning NevaClip with incorrect or misleading captions leads to random behavior, highlighting the significant impact of caption guidance in the decision-making process. These findings contribute to a better understanding of mechanisms that guide human attention and pave the way for more sophisticated computational approaches to scanpath prediction that can integrate direct top-down guidance of downstream tasks.
Experimental results from applying GPT-4 to an unpublished formal language

Can large language models be used to complete mathematical tasks that are traditionally performed either manually or with the aid of theorem provers? To answer this question, a state-of-the-art system, GPT-4, was provided with a concise natural language specification for a previously unpublished formal system and asked to complete a number of tasks, from stating function and type definitions to proving simple theorems and verifying user-supplied proofs. The system completed all tasks successfully, showed extensive domain knowledge, invented helpful new syntax and semantics, and exhibited generalization and inference abilities. So the answer seems to be: yes.
Motor antagonism dictates emergent dynamics in active double networks tuned by crosslinkers

The cytoskeleton relies on diverse populations of motors, filaments, and binding proteins acting in concert to enable non-equilibrium processes ranging from mitosis to chemotaxis. Its versatile reconfigurability, programmed by interactions between its constituents, make the cytoskeleton foundational active matter. Yet, current active matter endeavors are limited largely to single force-generating components acting on a single substrate, far from the composite cytoskeleton in cells. Here, we engineer actin-microtubule double-networks, driven by kinesin and myosin motors and tuned by crosslinkers, to ballistically restructure and flow with speeds that span three orders of magnitude depending on the composite formulation and time relative to the onset of motor activity. Differential dynamic microscopy analyses reveal that kinesin and myosin compete to delay the onset of acceleration and suppress discrete restructuring events, while passive crosslinking of either actin or microtubules has an opposite effect. Our minimal advection-diffusion model and spatial correlation analyses correlate these dynamics to structure, with motor antagonism suppressing reconfiguration and de-mixing, while crosslinking enhances clustering. Despite the rich formulation space and emergent formulation-dependent structures, the non-equilibrium dynamics across all networks and timescales can be organized into three classes: slow isotropic reorientation, fast directional flow, and multimode restructuring. Moreover, our mathematical model demonstrates that diverse structural motifs can arise simply from the interplay between motor-driven advection and frictional drag. These general features of our platform facilitate applicability to other active matter systems, and shed light on diverse ways that cytoskeletal components can cooperate or compete to enable wide-ranging cellular processes.
Understanding Differences in News Article Interaction Patterns on Facebook: Public vs. Private Sharing with Varying Bias and Reliability

The rapid growth of news dissemination and user engagement on social media has raised concerns about the influence and societal impact of biased and unreliable information. As a response to these concerns, a substantial body of research has been dedicated to understanding how users interact with different news. However, this research has primarily analyzed publicly shared posts. With a significant portion of engagement taking place within Facebook's private sphere, it is therefore important to also consider the private posts. In this paper, we present the first comprehensive comparison of the interaction patterns and depth of engagement between public and private posts of different types of news content shared on Facebook. To compare these patterns, we gathered and analyzed two complementary datasets: the first includes interaction data for all Facebook posts (private + public) referencing a manually labeled collection of over 19K news articles, while the second contains only interaction data for public posts tracked by CrowdTangle. As part of our methodology, we introduce several carefully designed data processing steps that address some critical aspects missed by prior works but that (through our iterative discussions and feedback with the CrowdTangle team) emerged as important to ensure fairness for this type of study. Our findings highlight significant disparities in interaction patterns across various news classes and spheres. For example, our statistical analysis demonstrates that users engage significantly more deeply with news in the private sphere compared to the public one, underscoring the pivotal role of considering both the public and private spheres of Facebook in future research. Beyond its scholarly impact, the findings of this study can benefit Facebook content moderators, regulators, and policymakers, contributing to a healthier online discourse.
Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models

Task arithmetic has recently emerged as a cost-effective and scalable approach to edit pre-trained models directly in weight space: By adding the fine-tuned weights of different tasks, the model's performance can be improved on these tasks, while negating them leads to task forgetting. Yet, our understanding of the effectiveness of task arithmetic and its underlying principles remains limited. We present a comprehensive study of task arithmetic in vision-language models and show that weight disentanglement is the crucial factor that makes it effective. This property arises during pre-training and manifests when distinct directions in weight space govern separate, localized regions in function space associated with the tasks. Notably, we show that fine-tuning models in their tangent space by linearizing them amplifies weight disentanglement. This leads to substantial performance improvements across multiple task arithmetic benchmarks and diverse models. Building on these findings, we provide theoretical and empirical analyses of the neural tangent kernel (NTK) of these models and establish a compelling link between task arithmetic and the spatial localization of the NTK eigenfunctions. Overall, our work uncovers novel insights into the fundamental mechanisms of task arithmetic and offers a more reliable and effective approach to edit pre-trained models through the NTK linearization.
ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain

The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill classification, job title classification, and de-identification. While some approaches have been developed that are specific to the job market domain, there is a lack of generalized, multilingual models and benchmarks for these tasks. In this study, we introduce a language model called ESCOXLM-R, based on XLM-R-large, which uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy, covering 27 languages. The pre-training objectives for ESCOXLM-R include dynamic masked language modeling and a novel additional objective for inducing multilingual taxonomical ESCO relations. We comprehensively evaluate the performance of ESCOXLM-R on 6 sequence labeling and 3 classification tasks in 4 languages and find that it achieves state-of-the-art results on 6 out of 9 datasets. Our analysis reveals that ESCOXLM-R performs better on short spans and outperforms XLM-R-large on entity-level and surface-level span-F1, likely due to ESCO containing short skill and occupation titles, and encoding information on the entity-level.
Specification and Runtime Checking of Derecho, A Protocol for Fast Replication for Cloud Services

Reliable distributed systems require replication and consensus among distributed processes to tolerate process and communication failures. Understanding and assuring the correctness of protocols for replication and consensus have been a significant challenge. This paper describes the precise specification and runtime checking of Derecho, a more recent, sophisticated protocol for fast replication and consensus for cloud services. A precise specification must fill in missing details and resolve ambiguities in English and pseudocode algorithm descriptions while also faithfully following the descriptions. To help check the correctness of the protocol, we also performed careful manual analysis and increasingly systematic runtime checking. We obtain a complete specification that is directly executable, and we discover and fix a number of issues in the pseudocode. These results were facilitated by the already detailed pseudocode of Derecho and made possible by using DistAlgo, a language that allows distributed algorithms to be easily and clearly expressed and directly executed.
Deep Discriminative to Kernel Density Graph for In- and Out-of-distribution Calibrated Inference

Deep discriminative approaches like random forests and deep neural networks have recently found applications in many important real-world scenarios. However, deploying these learning algorithms in safety-critical applications raises concerns, particularly when it comes to ensuring confidence calibration for both in-distribution and out-of-distribution data points. Many popular methods for in-distribution (ID) calibration, such as isotonic regression and Platt's sigmoidal regression, exhibit excellent ID calibration performance. However, these methods are not calibrated for the entire feature space, leading to overconfidence in the case of out-of-distribution (OOD) samples. On the other end of the spectrum, existing out-of-distribution (OOD) calibration methods generally exhibit poor in-distribution (ID) calibration. In this paper, we address ID and OOD calibration problems jointly. We leveraged the fact that deep models, including both random forests and deep-nets, learn internal representations which are unions of polytopes with affine activation functions to conceptualize them both as partitioning rules of the feature space. We replace the affine function in each polytope populated by the training data with a Gaussian kernel. We propose sufficient conditions for our proposed methods to be consistent estimators of the corresponding class conditional densities. Moreover, our experiments on both tabular and vision benchmarks show that the proposed approaches obtain well-calibrated posteriors while mostly preserving or improving the classification accuracy of the original algorithm for in-distribution region, and extrapolates beyond the training data to handle out-of-distribution inputs appropriately.
When Computing Power Network Meets Distributed Machine Learning: An Efficient Federated Split Learning Framework

In this paper, we advocate CPN-FedSL, a novel and flexible Federated Split Learning (FedSL) framework over Computing Power Network (CPN). We build a dedicated model to capture the basic settings and learning characteristics (e.g., training flow, latency and convergence). Based on this model, we introduce Resource Usage Effectiveness (RUE), a novel performance metric integrating training utility with system cost, and formulate a multivariate scheduling problem that maximizes RUE by comprehensively taking client admission, model partition, server selection, routing and bandwidth allocation into account (i.e., mixed-integer fractional programming). We design Refinery, an efficient approach that first linearizes the fractional objective and non-convex constraints, and then solves the transformed problem via a greedy based rounding algorithm in multiple iterations. Extensive evaluations corroborate that CPN-FedSL is superior to the standard and state-of-the-art learning frameworks (e.g., FedAvg and SplitFed), and besides Refinery is lightweight and significantly outperforms its variants and de facto heuristic methods under a variety of settings.
An Algorithm of Parameter Estimation for Frequency-hopping Signals

This paper presents an algorithm of parameter estimation for frequency-hopping(FH) signals.Having extracted the power spectrum density matrix changing according to time(PSDMCAT) of the cross-correlation function of FH signals,the method can estimate such parameters as hop-timing,hop-rate,dwell-time and frequency-set by analyzing the hops which are generated from PSDMCAT.The algorithm not only outperforms traditional methods via improving the signal noise ratio(SNR),but also doesn't need to know some preconditions such as hop-rate,dwell-time et al,and does not require the selection of a kernel or mother wavelet function.Meanwhile,for frequency-hopping signals whose hop-rate or dwell-time is alterable,the algorithm is also qualified.The field test shows the presented algorithm is effective.