OpenRaiser/Intern-Atlas huggingface dataset:
Action Spotting using Dense Detection Anchors Revisited: Submission to the SoccerNet Challenge 2022
Venue: arXiv cs.CV 2022
Year: 2022

This brief technical report describes our submission to the Action Spotting SoccerNet Challenge 2022. The challenge was part of the CVPR 2022 ActivityNet Workshop. Our submission was based on a recently proposed method which focuses on increasing temporal precision via a densely sampled set of detection anchors. Due to its emphasis on temporal precision, this approach had shown significant improvements in the tight average-mAP metric. Tight average-mAP was used as the evaluation criterion for the challenge, and is defined using small temporal evaluation tolerances, thus being more sensitive to small temporal errors. In order to further improve results, here we introduce small changes in the pre- and post-processing steps, and also combine different input feature types via late fusion. These changes brought improvements that helped us achieve the first place in the challenge and also led to a new state-of-the-art on SoccerNet's test set when using the dataset's standard experimental protocol. This report briefly reviews the action spotting method based on dense detection anchors, then focuses on the modifications introduced for the challenge. We also describe the experimental protocols and training procedures we used, and finally present our results.
EPG2S: Speech Generation and Speech Enhancement based on Electropalatography and Audio Signals using Multimodal Learning
Venue: arXiv cs.SD 2022
Year: 2022

Speech generation and enhancement based on articulatory movements facilitate communication when the scope of verbal communication is absent, e.g., in patients who have lost the ability to speak. Although various techniques have been proposed to this end, electropalatography (EPG), which is a monitoring technique that records contact between the tongue and hard palate during speech, has not been adequately explored. Herein, we propose a novel multimodal EPG-to-speech (EPG2S) system that utilizes EPG and speech signals for speech generation and enhancement. Different fusion strategies based on multiple combinations of EPG and noisy speech signals are examined, and the viability of the proposed method is investigated. Experimental results indicate that EPG2S achieves desirable speech generation outcomes based solely on EPG signals. Further, the addition of noisy speech signals is observed to improve quality and intelligibility. Additionally, EPG2S is observed to achieve high-quality speech enhancement based solely on audio signals, with the addition of EPG signals further improving the performance. The late fusion strategy is deemed to be the most effective approach for simultaneous speech generation and enhancement.
Challenges and Opportunities in Deep Reinforcement Learning with Graph Neural Networks: A Comprehensive review of Algorithms and Applications
Venue: arXiv cs.LG 2022
Year: 2022

Deep reinforcement learning (DRL) has empowered a variety of artificial intelligence fields, including pattern recognition, robotics, recommendation-systems, and gaming. Similarly, graph neural networks (GNN) have also demonstrated their superior performance in supervised learning for graph-structured data. In recent times, the fusion of GNN with DRL for graph-structured environments has attracted a lot of attention. This paper provides a comprehensive review of these hybrid works. These works can be classified into two categories: (1) algorithmic enhancement, where DRL and GNN complement each other for better utility; (2) application-specific enhancement, where DRL and GNN support each other. This fusion effectively addresses various complex problems in engineering and life sciences. Based on the review, we further analyze the applicability and benefits of fusing these two domains, especially in terms of increasing generalizability and reducing computational complexity. Finally, the key challenges in integrating DRL and GNN, and potential future research directions are highlighted, which will be of interest to the broader machine learning community.
Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos
Venue: arXiv cs.CV 2022
Year: 2022

Multimodal sentiment analysis in videos is a key task in many real-world applications, which usually requires integrating multimodal streams including visual, verbal and acoustic behaviors. To improve the robustness of multimodal fusion, some of the existing methods let different modalities communicate with each other and modal the crossmodal interaction via transformers. However, these methods only use the single-scale representations during the interaction but forget to exploit multi-scale representations that contain different levels of semantic information. As a result, the representations learned by transformers could be biased especially for unaligned multimodal data. In this paper, we propose a multi-scale cooperative multimodal transformer (MCMulT) architecture for multimodal sentiment analysis. On the whole, the "multi-scale" mechanism is capable of exploiting the different levels of semantic information of each modality which are used for fine-grained crossmodal interactions. Meanwhile, each modality learns its feature hierarchies via integrating the crossmodal interactions from multiple level features of its source modality. In this way, each pair of modalities progressively builds feature hierarchies respectively in a cooperative manner. The empirical results illustrate that our MCMulT model not only outperforms existing approaches on unaligned multimodal sequences but also has strong performance on aligned multimodal sequences.
DIALOG-22 RuATD Generated Text Detection
Venue: arXiv cs.CL 2022
Year: 2022

Text Generation Models (TGMs) succeed in creating text that matches human language style reasonably well. Detectors that can distinguish between TGM-generated text and human-written ones play an important role in preventing abuse of TGM. In this paper, we describe our pipeline for the two DIALOG-22 RuATD tasks: detecting generated text (binary task) and classification of which model was used to generate text (multiclass task). We achieved 1st place on the binary classification task with an accuracy score of 0.82995 on the private test set and 4th place on the multiclass classification task with an accuracy score of 0.62856 on the private test set. We proposed an ensemble method of different pre-trained models based on the attention mechanism.
Time Interval-enhanced Graph Neural Network for Shared-account Cross-domain Sequential Recommendation
Venue: arXiv cs.IR 2022
Year: 2022

Shared-account Cross-domain Sequential Recommendation (SCSR) task aims to recommend the next item via leveraging the mixed user behaviors in multiple domains. It is gaining immense research attention as more and more users tend to sign up on different platforms and share accounts with others to access domain-specific services. Existing works on SCSR mainly rely on mining sequential patterns via Recurrent Neural Network (RNN)-based models, which suffer from the following limitations: 1) RNN-based methods overwhelmingly target discovering sequential dependencies in single-user behaviors. They are not expressive enough to capture the relationships among multiple entities in SCSR. 2) All existing methods bridge two domains via knowledge transfer in the latent space, and ignore the explicit cross-domain graph structure. 3) None existing studies consider the time interval information among items, which is essential in the sequential recommendation for characterizing different items and learning discriminative representations for them. In this work, we propose a new graph-based solution, namely TiDA-GCN, to address the above challenges. Specifically, we first link users and items in each domain as a graph. Then, we devise a domain-aware graph convolution network to learn userspecific node representations. To fully account for users' domainspecific preferences on items, two effective attention mechanisms are further developed to selectively guide the message passing process. Moreover, to further enhance item- and account-level representation learning, we incorporate the time interval into the message passing, and design an account-aware self-attention module for learning items' interactive characteristics. Experiments demonstrate the superiority of our proposed method from various aspects.
Exploring Collaborative Game Play with Robots to Encourage Good Hand Hygiene Practises among Children
Venue: arXiv cs.RO 2022
Year: 2022

This paper presents the design, implementation, and evaluation of a novel collaborative educational game titled "Land of Hands", involving children and a customized social robot that we designed (HakshE). Through this gaming platform, we aim to teach proper hand hygiene practises to children and explore the extent of interactions that take place between a pro-social robot and children in such a setting. We blended gamification with Computers as Social Actors (CASA) paradigm to model the robot as a social actor or a fellow player in the game. The game was developed using Godot's 2D engine and Alice 3. In this study, 32 participants played the game online through a video teleconferencing platform Zoom. To understand the influence a pro-social robot's nudges has on children's interactions, we split our study into two conditions: With-Nudges and Without-Nudges. Detailed analysis of rubrics and video analyses of children's interactions show that our platform helped children learn good hand hygiene practises. We also found that using a pro-social robot creates enjoyable interactions and greater social engagement between the children and the robot although learning itself wasn't influenced by the pro-sociality of the robot.
Deep Neural Imputation: A Framework for Recovering Incomplete Brain Recordings
Venue: arXiv cs.LG 2022
Year: 2022

Neuroscientists and neuroengineers have long relied on multielectrode neural recordings to study the brain. However, in a typical experiment, many factors corrupt neural recordings from individual electrodes, including electrical noise, movement artifacts, and faulty manufacturing. Currently, common practice is to discard these corrupted recordings, reducing already limited data that is difficult to collect. To address this challenge, we propose Deep Neural Imputation (DNI), a framework to recover missing values from electrodes by learning from data collected across spatial locations, days, and participants. We explore our framework with a linear nearest-neighbor approach and two deep generative autoencoders, demonstrating DNI's flexibility. One deep autoencoder models participants individually, while the other extends this architecture to model many participants jointly. We evaluate our models across 12 human participants implanted with multielectrode intracranial electrocorticography arrays; participants had no explicit task and behaved naturally across hundreds of recording hours. We show that DNI recovers not only time series but also frequency content, and further establish DNI's practical value by recovering significant performance on a scientifically-relevant downstream neural decoding task.
Volumetric Supervised Contrastive Learning for Seismic Semantic Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

In seismic interpretation, pixel-level labels of various rock structures can be time-consuming and expensive to obtain. As a result, there oftentimes exists a non-trivial quantity of unlabeled data that is left unused simply because traditional deep learning methods rely on access to fully labeled volumes. To rectify this problem, contrastive learning approaches have been proposed that use a self-supervised methodology in order to learn useful representations from unlabeled data. However, traditional contrastive learning approaches are based on assumptions from the domain of natural images that do not make use of seismic context. In order to incorporate this context within contrastive learning, we propose a novel positive pair selection strategy based on the position of slices within a seismic volume. We show that the learnt representations from our method out-perform a state of the art contrastive learning methodology in a semantic segmentation task.
Adversarial Privacy Protection on Speech Enhancement
Venue: arXiv cs.SD 2022
Year: 2022

Speech is easily leaked imperceptibly, such as being recorded by mobile phones in different situations. Private content in speech may be maliciously extracted through speech enhancement technology. Speech enhancement technology has developed rapidly along with deep neural networks (DNNs), but adversarial examples can cause DNNs to fail. In this work, we propose an adversarial method to degrade speech enhancement systems. Experimental results show that generated adversarial examples can erase most content information in original examples or replace it with target speech content through speech enhancement. The word error rate (WER) between an enhanced original example and enhanced adversarial example recognition result can reach 89.0%. WER of target attack between enhanced adversarial example and target example is low to 33.75% . Adversarial perturbation can bring the rate of change to the original example to more than 1.4430. This work can prevent the malicious extraction of speech.
Downlink Spectral Efficiency of Massive MIMO with Dual-Polarized Antennas
Venue: arXiv cs.IT 2022
Year: 2022

This paper considers the downlink of a single-cell massive MIMO (multiple-input multiple-output) system with dual-polarized antennas at both the base station and users. We consider a channel model that takes into account several practical aspects that arise when utilizing dual polarization, such as channel cross-polar discrimination (XPD) and cross-polar receive and transmit correlations (XPC). We derive the statistical properties of the minimum mean squared error (MMSE) channel estimator for this model. Using these estimates for maximum ratio precoding, a rigorous closed-form downlink spectral efficiency (SE) expression is derived. We compare the SEs achieved in dual-polarized and uni-polarized setups numerically and evaluate the impact of XPD on the downlink SE.
Noisy Learning for Neural ODEs Acts as a Robustness Locus Widening
Venue: arXiv cs.LG 2022
Year: 2022

We investigate the problems and challenges of evaluating the robustness of Differential Equation-based (DE) networks against synthetic distribution shifts. We propose a novel and simple accuracy metric which can be used to evaluate intrinsic robustness and to validate dataset corruption simulators. We also propose methodology recommendations, destined for evaluating the many faces of neural DEs' robustness and for comparing them with their discrete counterparts rigorously. We then use this criteria to evaluate a cheap data augmentation technique as a reliable way for demonstrating the natural robustness of neural ODEs against simulated image corruptions across multiple datasets.
On the Surprising Behaviour of node2vec
Venue: arXiv cs.LG 2022
Year: 2022

Graph embedding techniques are a staple of modern graph learning research. When using embeddings for downstream tasks such as classification, information about their stability and robustness, i.e., their susceptibility to sources of noise, stochastic effects, or specific parameter choices, becomes increasingly important. As one of the most prominent graph embedding schemes, we focus on node2vec and analyse its embedding quality from multiple perspectives. Our findings indicate that embedding quality is unstable with respect to parameter choices, and we propose strategies to remedy this in practice.
Gradient Descent for Low-Rank Functions
Venue: arXiv cs.LG 2022
Year: 2022

Several recent empirical studies demonstrate that important machine learning tasks, e.g., training deep neural networks, exhibit low-rank structure, where the loss function varies significantly in only a few directions of the input space. In this paper, we leverage such low-rank structure to reduce the high computational cost of canonical gradient-based methods such as gradient descent (GD). Our proposed \emph{Low-Rank Gradient Descent} (LRGD) algorithm finds an $ε$-approximate stationary point of a $p$-dimensional function by first identifying $r \leq p$ significant directions, and then estimating the true $p$-dimensional gradient at every iteration by computing directional derivatives only along those $r$ directions. We establish that the "directional oracle complexities" of LRGD for strongly convex and non-convex objective functions are $\mathcal{O}(r \log(1/ε) + rp)$ and $\mathcal{O}(r/ε^2 + rp)$, respectively. When $r \ll p$, these complexities are smaller than the known complexities of $\mathcal{O}(p \log(1/ε))$ and $\mathcal{O}(p/ε^2)$ of {\gd} in the strongly convex and non-convex settings, respectively. Thus, LRGD significantly reduces the computational cost of gradient-based methods for sufficiently low-rank functions. In the course of our analysis, we also formally define and characterize the classes of exact and approximately low-rank functions.
ProGNNosis: A Data-driven Model to Predict GNN Computation Time Using Graph Metrics
Venue: arXiv cs.LG 2022
Year: 2022

Graph Neural Networks (GNN) show great promise in problems dealing with graph-structured data. One of the unique points of GNNs is their flexibility to adapt to multiple problems, which not only leads to wide applicability, but also poses important challenges when finding the best model or acceleration technique for a particular problem. An example of such challenges resides in the fact that the accuracy or effectiveness of a GNN model or acceleration technique generally depends on the structure of the underlying graph. In this paper, in an attempt to address the problem of graph-dependent acceleration, we propose ProGNNosis, a data-driven model that can predict the GNN training time of a given GNN model running over a graph of arbitrary characteristics by inspecting the input graph metrics. Such prediction is made based on a regression that was previously trained offline using a diverse synthetic graph dataset. In practice, our method allows making informed decisions on which design to use for a specific problem. In the paper, the methodology to build ProGNNosis is defined and applied for a specific use case, where it helps to decide which graph representation is better. Our results show that ProGNNosis helps achieve an average speedup of 1.22X over randomly selecting a graph representation in multiple widely used GNN models such as GCN, GIN, GAT, or GraphSAGE.
Securing Automotive Architectures with Named Data Networking
Venue: arXiv cs.NI 2022
Year: 2022

As in-vehicle communication becomes more complex, the automotive community is exploring various architectural options such as centralized and zonal architectures for their numerous benefits. Zonal architecture reduces the wiring cost by physically locating related operations and ECUs near their intended functions and the number of physical ECUs through function consolidation. Centralized architectures consolidate the number of ECUs into few, powerful compute units. Common characteristics of these architectures include the need for high-bandwidth communication and security, which have been elusive with standard automotive architectures. Further, as automotive communication technologies evolve, it is also likely that multiple link-layer technologies such as CAN and Automotive Ethernet will co-exist. These alternative architectures promise to integrate these diverse sets of technologies. However, architectures that allow such co-existence have not been adequately explored. In this work we explore a new network architecture called Named Data Networking (NDN) to achieve multiple goals: provide a foundational security infrastructure and bridge different link layer protocols such as CAN, LIN, and automotive Ethernet into a unified communication system. We created a proof-of-concept bench-top testbed using CAN HATS and Raspberry PIs that replay real traffic over CAN and Ethernet to demonstrate how NDN can provide a secure, high-speed bridge between different automotive link layers. We also show how NDN can support communication between centralized or zonal high-power compute components. Security is achieved through digitally signing all Data packets between these components, preventing unauthorized ECUs from injecting arbitrary data into the network. We also demonstrate NDN's ability to prevent DoS and replay attacks between different network segments connected through NDN.
Planning and Formulations in Pursuit-Evasion: Keep-away Games and Their Strategies
Venue: arXiv cs.RO 2022
Year: 2022

We study a pursuit-evasion problem which can be viewed as an extension of the keep-away game. In the game, pursuer(s) will attempt to intersect or catch the evader, while the evader can visit a fixed set of locations, which we denote as the anchors. These anchors may or may not be stationary. When the velocity of the pursuers is limited and considered low compared to the evaders, we are interested in whether a winning strategy exists for the pursuers or the evaders, or the game will draw. When the anchors are stationary, we show an algorithm that can help answer the above question. The primary motivation for this study is to explore the boundaries between kinematic and dynamic constraints. In particular, whether the solution of the kinematic problem can be used to speed up the search for the problems with dynamic constraints and how to discretize the problem to utilize such relations best. In this work, we show that a geometric branch-and-bound type of approach can be used to solve the stationary anchor problem, and the approach and the solution can be extended to solve the dynamic problem where the pursuers have dynamic constraints, including velocity and acceleration bounds.
User Customizable and Robust Geo-Indistinguishability for Location Privacy
Venue: arXiv cs.CR 2022
Year: 2022

Location obfuscation functions generated by existing systems for ensuring location privacy are monolithic and do not allow users to customize their obfuscation range. This can lead to the user being mapped in undesirable locations (e.g., shady neighborhoods) to the location-requesting services. Modifying the obfuscation function generated by a centralized server on the user side can result in poor privacy as the original function is not robust against such updates. Users themselves might find it challenging to understand the parameters involved in obfuscation mechanisms (e.g., obfuscation range and granularity of location representation) and therefore struggle to set realistic trade-offs between privacy, utility, and customization. In this paper, we propose a new framework called, CORGI, i.e., CustOmizable Robust Geo-Indistinguishability, which generates location obfuscation functions that are robust against user customization while providing strong privacy guarantees based on the Geo-Indistinguishability paradigm. CORGI utilizes a tree representation of a given region to assist users in specifying their privacy and customization requirements. The server side of CORGI takes these requirements as inputs and generates an obfuscation function that satisfies Geo-Indistinguishability requirements and is robust against customization on the user side. The obfuscation function is returned to the user who can then choose to update the obfuscation function (e.g., obfuscation range, granularity of location representation). The experimental results on a real dataset demonstrate that CORGI can efficiently generate obfuscation matrices that are more robust to the customization by users.