OpenRaiser/Intern-Atlas huggingface dataset:
Adaptive Affinity for Associations in Multi-Target Multi-Camera Tracking
Venue: arXiv cs.CV 2021
Year: 2021

Data associations in multi-target multi-camera tracking (MTMCT) usually estimate affinity directly from re-identification (re-ID) feature distances. However, we argue that it might not be the best choice given the difference in matching scopes between re-ID and MTMCT problems. Re-ID systems focus on global matching, which retrieves targets from all cameras and all times. In contrast, data association in tracking is a local matching problem, since its candidates only come from neighboring locations and time frames. In this paper, we design experiments to verify such misfit between global re-ID feature distances and local matching in tracking, and propose a simple yet effective approach to adapt affinity estimations to corresponding matching scopes in MTMCT. Instead of trying to deal with all appearance changes, we tailor the affinity metric to specialize in ones that might emerge during data associations. To this end, we introduce a new data sampling scheme with temporal windows originally used for data associations in tracking. Minimizing the mismatch, the adaptive affinity module brings significant improvements over global re-ID distance, and produces competitive performance on CityFlow and DukeMTMC datasets.
Learning to track environment state via predictive autoencoding
Venue: arXiv cs.LG 2021
Year: 2021

This work introduces a neural architecture for learning forward models of stochastic environments. The task is achieved solely through learning from temporal unstructured observations in the form of images. Once trained, the model allows for tracking of the environment state in the presence of noise or with new percepts arriving intermittently. Additionally, the state estimate can be propagated in observation-blind mode, thus allowing for long-term predictions. The network can output both expectation over future observations and samples from belief distribution. The resulting functionalities are similar to those of a Particle Filter (PF). The architecture is evaluated in an environment where we simulate objects moving. As the forward and sensor models are available, we implement a PF to gauge the quality of the models learnt from the data.
Do Answers to Boolean Questions Need Explanations? Yes
Venue: arXiv cs.CL 2021
Year: 2021

Existing datasets that contain boolean questions, such as BoolQ and TYDI QA , provide the user with a YES/NO response to the question. However, a one word response is not sufficient for an explainable system. We promote explainability by releasing a new set of annotations marking the evidence in existing TyDi QA and BoolQ datasets. We show that our annotations can be used to train a model that extracts improved evidence spans compared to models that rely on existing resources. We confirm our findings with a user study which shows that our extracted evidence spans enhance the user experience. We also provide further insight into the challenges of answering boolean questions, such as passages containing conflicting YES and NO answers, and varying degrees of relevance of the predicted evidence.
EDAssistant: Supporting Exploratory Data Analysis in Computational Notebooks with In-Situ Code Search and Recommendation
Venue: arXiv cs.HC 2021
Year: 2021

Using computational notebooks (e.g., Jupyter Notebook), data scientists rationalize their exploratory data analysis (EDA) based on their prior experience and external knowledge such as online examples. For novices or data scientists who lack specific knowledge about the dataset or problem to investigate, effectively obtaining and understanding the external information is critical to carry out EDA. This paper presents EDAssistant, a JupyterLab extension that supports EDA with in-situ search of example notebooks and recommendation of useful APIs, powered by novel interactive visualization of search results. The code search and recommendation are enabled by state-of-the-art machine learning models, trained on a large corpus of EDA notebooks collected online. A user study is conducted to investigate both EDAssistant and data scientists' current practice (i.e., using external search engines). The results demonstrate the effectiveness and usefulness of EDAssistant, and participants appreciated its smooth and in-context support of EDA. We also report several design implications regarding code recommendation tools.
Decoding Continual Muscle Movements Related to Complex Hand Grasping from EEG Signals
Venue: arXiv cs.HC 2021
Year: 2021

Brain-computer interface (BCI) is a practical pathway to interpret users' intentions by decoding motor execution (ME) or motor imagery (MI) from electroencephalogram (EEG) signals. However, developing a BCI system driven by ME or MI is challenging, particularly in the case of containing continual and compound muscles movements. This study analyzes three grasping actions from EEG under both ME and MI paradigms. We also investigate the classification performance in offline and pseudo-online experiments. We propose a novel approach that uses muscle activity pattern (MAP) images for the convolutional neural network (CNN) to improve classification accuracy. We record the EEG and electromyogram (EMG) signals simultaneously and create the MAP images by decoding both signals to estimate specific hand grasping. As a result, we obtained an average classification accuracy of 63.6($\pm$6.7)% in ME and 45.8($\pm$4.4)% in MI across all fifteen subjects for four classes. Also, we performed pseudo-online experiments and obtained classification accuracies of 60.5($\pm$8.4)% in ME and 42.7($\pm$6.8)% in MI. The proposed method MAP-CNN, shows stable classification performance, even in the pseudo-online experiment. We expect that MAP-CNN could be used in various BCI applications in the future.
Transcoded Video Restoration by Temporal Spatial Auxiliary Network
Venue: arXiv cs.CV 2021
Year: 2021

In most video platforms, such as Youtube, and TikTok, the played videos usually have undergone multiple video encodings such as hardware encoding by recording devices, software encoding by video editing apps, and single/multiple video transcoding by video application servers. Previous works in compressed video restoration typically assume the compression artifacts are caused by one-time encoding. Thus, the derived solution usually does not work very well in practice. In this paper, we propose a new method, temporal spatial auxiliary network (TSAN), for transcoded video restoration. Our method considers the unique traits between video encoding and transcoding, and we consider the initial shallow encoded videos as the intermediate labels to assist the network to conduct self-supervised attention training. In addition, we employ adjacent multi-frame information and propose the temporal deformable alignment and pyramidal spatial fusion for transcoded video restoration. The experimental results demonstrate that the performance of the proposed method is superior to that of the previous techniques. The code is available at https://github.com/icecherylXuli/TSAN.
Modality-Aware Triplet Hard Mining for Zero-shot Sketch-Based Image Retrieval
Venue: arXiv cs.CV 2021
Year: 2021

This paper tackles the Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) problem from the viewpoint of cross-modality metric learning. This task has two characteristics: 1) the zero-shot setting requires a metric space with good within-class compactness and the between-class discrepancy for recognizing the novel classes and 2) the sketch query and the photo gallery are in different modalities. The metric learning viewpoint benefits ZS-SBIR from two aspects. First, it facilitates improvement through recent good practices in deep metric learning (DML). By combining two fundamental learning approaches in DML, e.g., classification training and pairwise training, we set up a strong baseline for ZS-SBIR. Without bells and whistles, this baseline achieves competitive retrieval accuracy. Second, it provides an insight that properly suppressing the modality gap is critical. To this end, we design a novel method named Modality-Aware Triplet Hard Mining (MATHM). MATHM enhances the baseline with three types of pairwise learning, e.g., a cross-modality sample pair, a within-modality sample pair, and their combination.\We also design an adaptive weighting method to balance these three components during training dynamically. Experimental results confirm that MATHM brings another round of significant improvement based on the strong baseline and sets up new state-of-the-art performance. For example, on the TU-Berlin dataset, we achieve 47.88+2.94% mAP@all and 58.28+2.34% Prec@100. Code will be publicly available at: https://github.com/huangzongheng/MATHM.
Learning Submodular Objectives for Team Environmental Monitoring
Venue: arXiv cs.RO 2021
Year: 2021

In this paper, we study the well-known team orienteering problem where a fleet of robots collects rewards by visiting locations. Usually, the rewards are assumed to be known to the robots; however, in applications such as environmental monitoring or scene reconstruction, the rewards are often subjective and specifying them is challenging. We propose a framework to learn the unknown preferences of the user by presenting alternative solutions to them, and the user provides a ranking on the proposed alternative solutions. We consider the two cases for the user: 1) a deterministic user which provides the optimal ranking for the alternative solutions, and 2) a noisy user which provides the optimal ranking according to an unknown probability distribution. For the deterministic user we propose a framework to minimize a bound on the maximum deviation from the optimal solution, namely regret. We adapt the approach to capture the noisy user and minimize the expected regret. Finally, we demonstrate the importance of learning user preferences and the performance of the proposed methods in an extensive set of experimental results using real world datasets for environmental monitoring problems.
Service Oriented Architecture in Enterprise Application
Venue: arXiv cs.SE 2021
Year: 2021

At present organizations try to achieve competitive advantages using information technology (IT). Organizations not only use Information technology to manage their internal operations but also to collaborate with their customers and suppliers. For these organizations use enterprise applications. Also, organizations expect IT to address their shifting needs in on demand environment. So currently IT face challenges in integrating the various system into a function that can address the organization's on-demand needs and span over the organizational boundaries. Now Service-Oriented Architecture (SOA) imagine as an architectural framework that addresses the issues on previous enterprise applications. This paper represents the Service-Oriented Architecture in Enterprise Applications. The enterprise application gives a brief idea about enterprise applications. Afterward, this paper addresses the main problems faced by enterprise applications on the evolution of enterprise applications and the needs of service-oriented architecture in enterprise applications. Afterward, discuss Service Oriented Architecture and challenges on Service Oriented Architecture.
Visually Guided UGV for Autonomous Mobile Manipulation in Dynamic and Unstructured GPS Denied Environments
Venue: arXiv cs.RO 2021
Year: 2021

A robotic solution for the unmanned ground vehicles (UGVs) to execute the highly complex task of object manipulation in an autonomous mode is presented. This paper primarily focuses on developing an autonomous robotic system capable of assembling elementary blocks to build the large 3D structures in GPS-denied environments. The key contributions of this system paper are i) Designing of a deep learning-based unified multi-task visual perception system for object detection, part-detection, instance segmentation, and tracking, ii) an electromagnetic gripper design for robust grasping, and iii) system integration in which multiple system components are integrated to develop an optimized software stack. The entire mechatronic and algorithmic design of UGV for the above application is detailed in this work. The performance and efficacy of the overall system are reported through several rigorous experiments.
Meshless Monte Carlo Radiation Transfer Method for Curved Geometries using Signed Distance Functions
Venue: arXiv cs.GR 2021
Year: 2021

Significance: Monte Carlo radiation transfer (MCRT) is the gold standard of modeling light transport in turbid media. Typical MCRT models use voxels or meshes to approximate experimental geometry. A voxel based geometry does not allow for the accurate modeling of smooth curved surfaces, such as may be found in biological systems or food and drink packaging. Aim: We present our algorithm which we term signedMCRT (sMCRT), a new geometry-based method which uses signed distance functions (SDF) to represent the geometry of the model. SDFs are capable of modeling smooth curved surfaces accurately whilst also modeling complex geometries. Approach: We show that using SDFs to represent the problem's geometry is more accurate and can be faster than voxel based methods. sMCRT, can easily be incorporated into existing voxel based models. Results: sMCRT is validated against theoretical expressions, and other voxel based MCRT codes. We show that sMCRT can accurately model arbitrary complex geometries such as microvascular vessel network using SDFs. In comparison to the current state-of-the-art in MCRT methods specifically for curved surfaces, sMCRT is up-to forty-five times more accurate. Conclusions: sMCRT is a highly accurate, fast MCRT method that outperforms comparable voxel based models due to its ability to model smooth curved surfaces. sMCRT is up-to three times faster than a voxel model for equivalent scenarios. sMCRT is publicly available at https://github.com/lewisfish/signedMCRT
Towards Controllable Agent in MOBA Games with Generative Modeling
Venue: arXiv cs.LG 2021
Year: 2021

We propose novel methods to develop action controllable agent that behaves like a human and has the ability to align with human players in Multiplayer Online Battle Arena (MOBA) games. By modeling the control problem as an action generation process, we devise a deep latent alignment neural network model for training agent, and a corresponding sampling algorithm for controlling an agent's action. Particularly, we propose deterministic and stochastic attention implementations of the core latent alignment model. Both simulated and online experiments in the game Honor of Kings demonstrate the efficacy of the proposed methods.
Text Gestalt: Stroke-Aware Scene Text Image Super-Resolution
Venue: arXiv cs.CV 2021
Year: 2021

In the last decade, the blossom of deep learning has witnessed the rapid development of scene text recognition. However, the recognition of low-resolution scene text images remains a challenge. Even though some super-resolution methods have been proposed to tackle this problem, they usually treat text images as general images while ignoring the fact that the visual quality of strokes (the atomic unit of text) plays an essential role for text recognition. According to Gestalt Psychology, humans are capable of composing parts of details into the most similar objects guided by prior knowledge. Likewise, when humans observe a low-resolution text image, they will inherently use partial stroke-level details to recover the appearance of holistic characters. Inspired by Gestalt Psychology, we put forward a Stroke-Aware Scene Text Image Super-Resolution method containing a Stroke-Focused Module (SFM) to concentrate on stroke-level internal structures of characters in text images. Specifically, we attempt to design rules for decomposing English characters and digits at stroke-level, then pre-train a text recognizer to provide stroke-level attention maps as positional clues with the purpose of controlling the consistency between the generated super-resolution image and high-resolution ground truth. The extensive experimental results validate that the proposed method can indeed generate more distinguishable images on TextZoom and manually constructed Chinese character dataset Degraded-IC13. Furthermore, since the proposed SFM is only used to provide stroke-level guidance when training, it will not bring any time overhead during the test phase. Code is available at https://github.com/FudanVI/FudanOCR/tree/main/text-gestalt.
A Factorization Approach for Motor Imagery Classification
Venue: arXiv cs.CV 2021
Year: 2021

Brain-computer interface uses brain signals to communicate with external devices without actual control. Many studies have been conducted to classify motor imagery based on machine learning. However, classifying imagery data with sparse spatial characteristics, such as single-arm motor imagery, remains a challenge. In this paper, we proposed a method to factorize EEG signals into two groups to classify motor imagery even if spatial features are sparse. Based on adversarial learning, we focused on extracting common features of EEG signals which are robust to noise and extracting only signal features. In addition, class-specific features were extracted which are specialized for class classification. Finally, the proposed method classifies the classes by representing the features of the two groups as one embedding space. Through experiments, we confirmed the feasibility that extracting features into two groups is advantageous for datasets that contain sparse spatial features.
Quantitative analysis of visual representation of sign elements in COVID-19 context
Venue: arXiv cs.CV 2021
Year: 2021

Representation is the way in which human beings re-present the reality of what is happening, both externally and internally. Thus, visual representation as a means of communication uses elements to build a narrative, just as spoken and written language do. We propose using computer analysis to perform a quantitative analysis of the elements used in the visual creations that have been produced in reference to the epidemic, using the images compiled in The Covid Art Museum's Instagram account to analyze the different elements used to represent subjective experiences with regard to a global event. This process has been carried out with techniques based on machine learning to detect objects in the images so that the algorithm can be capable of learning and detecting the objects contained in each study image. This research reveals that the elements that are repeated in images to create narratives and the relations of association that are established in the sample, concluding that, despite the subjectivity that all creation entails, there are certain parameters of shared and reduced decisions when it comes to selecting objects to be included in visual representations
Blockchain as an IoT intermediary
Venue: arXiv cs.CR 2021
Year: 2021

Blockchain technology provides a private, secure, transparent decentralized exchange of data. Also, blockchain is not limited to a particular area, but it has a wide range of applications and can be integrated into a variety of Internet interactive systems. For example, the Internet of Things (IoT), supply chain tracking, Electronic Health Records (EHR), digital forensics, identity management, trustless payments, and other key business elements will all benefit from its implementation. Next layer solutions such as Ethereum 2.0, Polkadot, Cardano, and other Web 3.0 technologies provide developers versatility. Moreover, these platforms utilize smart contracts which are similar to standard, traditionalized software during development but offer key utilities to end-users such as online wallets, secure data with transparent rules. Blockchain is receiving a lot of attention in educational technology (EduTech) as it aims to achieve a more transparent and multipurpose educational system. In addition to smart contract technology which defines how data should be registered, gathered and processed, blockchain can be used as an IoT intermediary for mobile usage. Therefore, we implemented an educational learning platform powered by blockchain technology to examine feasibility in industry and academic environment. In essence, this is a web application which is adapted to mobile platform and connected to blockchain for crucial data exchanges. In this paper we want to emphasize the potential of blockchain technology in multiple sectors as well as the need to really understand the underlying principles which are allowing disruptability of traditional centralized software solutions.
Expert and Crowd-Guided Affect Annotation and Prediction
Venue: arXiv cs.MM 2021
Year: 2021

We employ crowdsourcing to acquire time-continuous affective annotations for movie clips, and refine noisy models trained from these crowd annotations incorporating expert information within a Multi-task Learning (MTL) framework. We propose a novel \textbf{e}xpert \textbf{g}uided MTL (EG-MTL) algorithm, which minimizes the loss with respect to both crowd and expert labels to learn a set of weights corresponding to each movie clip for which crowd annotations are acquired. We employ EG-MTL to solve two problems, namely, \textbf{\texttt{P1}}: where dynamic annotations acquired from both experts and crowdworkers for the \textbf{Validation} set are used to train a regression model with audio-visual clip descriptors as features, and predict dynamic arousal and valence levels on 5--15 second snippets derived from the clips; and \textbf{\texttt{P2}}: where a classification model trained on the \textbf{Validation} set using dynamic crowd and expert annotations (as features) and static affective clip labels is used for binary emotion recognition on the \textbf{Evaluation} set for which only dynamic crowd annotations are available. Observed experimental results confirm the effectiveness of the EG-MTL algorithm, which is reflected via improved arousal and valence estimation for \textbf{\texttt{P1}}, and higher recognition accuracy for \textbf{\texttt{P2}}.
Insta-VAX: A Multimodal Benchmark for Anti-Vaccine and Misinformation Posts Detection on Social Media
Venue: arXiv cs.CL 2021
Year: 2021

Sharing of anti-vaccine posts on social media, including misinformation posts, has been shown to create confusion and reduce the publics confidence in vaccines, leading to vaccine hesitancy and resistance. Recent years have witnessed the fast rise of such anti-vaccine posts in a variety of linguistic and visual forms in online networks, posing a great challenge for effective content moderation and tracking. Extending previous work on leveraging textual information to understand vaccine information, this paper presents Insta-VAX, a new multi-modal dataset consisting of a sample of 64,957 Instagram posts related to human vaccines. We applied a crowdsourced annotation procedure verified by two trained expert judges to this dataset. We then bench-marked several state-of-the-art NLP and computer vision classifiers to detect whether the posts show anti-vaccine attitude and whether they contain misinformation. Extensive experiments and analyses demonstrate the multimodal models can classify the posts more accurately than the uni-modal models, but still need improvement especially on visual context understanding and external knowledge cooperation. The dataset and classifiers contribute to monitoring and tracking of vaccine discussions for social scientific and public health efforts in combating the problem of vaccine misinformation.