OpenRaiser/Intern-Atlas huggingface dataset:
IVT: An End-to-End Instance-guided Video Transformer for 3D Pose Estimation
Venue: arXiv cs.CV 2022
Year: 2022

Video 3D human pose estimation aims to localize the 3D coordinates of human joints from videos. Recent transformer-based approaches focus on capturing the spatiotemporal information from sequential 2D poses, which cannot model the contextual depth feature effectively since the visual depth features are lost in the step of 2D pose estimation. In this paper, we simplify the paradigm into an end-to-end framework, Instance-guided Video Transformer (IVT), which enables learning spatiotemporal contextual depth information from visual features effectively and predicts 3D poses directly from video frames. In particular, we firstly formulate video frames as a series of instance-guided tokens and each token is in charge of predicting the 3D pose of a human instance. These tokens contain body structure information since they are extracted by the guidance of joint offsets from the human center to the corresponding body joints. Then, these tokens are sent into IVT for learning spatiotemporal contextual depth. In addition, we propose a cross-scale instance-guided attention mechanism to handle the variational scales among multiple persons. Finally, the 3D poses of each person are decoded from instance-guided tokens by coordinate regression. Experiments on three widely-used 3D pose estimation benchmarks show that the proposed IVT achieves state-of-the-art performances.
Imagining Future Digital Assistants at Work: A Study of Task Management Needs
Venue: arXiv cs.HC 2022
Year: 2022

Digital Assistants (DAs) can support workers in the workplace and beyond. However, target user needs are not fully understood, and the functions that workers would ideally want a DA to support require further study. A richer understanding of worker needs could help inform the design of future DAs. We investigate user needs of future workplace DAs using data from a user study of 40 workers over a four-week period. Our qualitative analysis confirms existing research and generates new insight on the role of DAs in managing people's time, tasks, and information. Placing these insights in relation to quantitative analysis of self-reported task data, we highlight how different occupation roles require DAs to take varied approaches to these domains and the effect of task characteristics on the imagined features. Our findings have implications for the design of future DAs in work settings, and we offer some recommendations for reduction to practice.
"All of them claim to be the best": Multi-perspective study of VPN users and VPN providers
Venue: arXiv cs.CR 2022
Year: 2022

As more users adopt VPNs for a variety of reasons, it is important to develop empirical knowledge of their needs and mental models of what a VPN offers. Moreover, studying VPN users alone is not enough because, by using a VPN, a user essentially transfers trust, say from their network provider, onto the VPN provider. To that end, we are the first to study the VPN ecosystem from both the users' and the providers' perspectives. In this paper, we conduct a quantitative survey of 1,252 VPN users in the U.S. and qualitative interviews of nine providers to answer several research questions regarding the motivations, needs, threat model, and mental model of users, and the key challenges and insights from VPN providers. We create novel insights by augmenting our multi-perspective results, and highlight cases where the user and provider perspectives are misaligned. Alarmingly, we find that users rely on and trust VPN review sites, but VPN providers shed light on how these sites are mostly motivated by money. Worryingly, we find that users have flawed mental models about the protection VPNs provide, and about data collected by VPNs. We present actionable recommendations for technologists and security and privacy advocates by identifying potential areas on which to focus efforts and improve the VPN ecosystem.
Follow Me: Conversation Planning for Target-driven Recommendation Dialogue Systems
Venue: arXiv cs.CL 2022
Year: 2022

Recommendation dialogue systems aim to build social bonds with users and provide high-quality recommendations. This paper pushes forward towards a promising paradigm called target-driven recommendation dialogue systems, which is highly desired yet under-explored. We focus on how to naturally lead users to accept the designated targets gradually through conversations. To this end, we propose a Target-driven Conversation Planning (TCP) framework to plan a sequence of dialogue actions and topics, driving the system to transit between different conversation stages proactively. We then apply our TCP with planned content to guide dialogue generation. Experimental results show that our conversation planning significantly improves the performance of target-driven recommendation dialogue systems.
MetaEmu: An Architecture Agnostic Rehosting Framework for Automotive Firmware
Venue: arXiv cs.CR 2022
Year: 2022

In this paper we present MetaEmu, an architecture-agnostic emulator synthesizer geared towards rehosting and security analysis of automotive firmware. MetaEmu improves over existing rehosting environments in two ways: Firstly, it solves the hitherto open-problem of a lack of generic Virtual Execution Environments (VXEs) for rehosting by synthesizing processor simulators from Ghidra's language definitions. In doing so, MetaEmu can simulate any processor supported by a vast and growing library of open-source definitions. In MetaEmu, we use a specification-based approach to cover peripherals, execution models, and analyses, which allows our framework to be easily extended. Secondly, MetaEmu can rehost and analyze multiple targets, each of different architecture, simultaneously, and share analysis facts between each target's analysis environment, a technique we call inter-device analysis. We show that the flexibility afforded by our approach does not lead to a performance trade-off -- MetaEmu lifts rehosted firmware to an optimized intermediate representation, and provides performance comparable to existing emulation tools, such as Unicorn. Our evaluation spans five different architectures, bare-metal and RTOS-based firmware, and three kinds of automotive Electronic Control Unit (ECU) from four distinct vendors -- none of which can be rehosted or emulated by current tools, due to lack of processor support. Further, we show how MetaEmu enables a diverse set of analyses by implementing a fuzzer, a symbolic executor for solving peripheral access checks, a CAN ID reverse engineering tool, and an inter-device coverage tracker.
Analysis of Temporal Robustness in Massive Machine Type Communications
Venue: arXiv cs.NI 2022
Year: 2022

The evolution of fifth generation (5G) networks needs to support the latest use cases, which demand robust network connectivity for the collaborative performance of the network agents, like multi-robot systems and vehicle to anything (V2X) communication. Unfortunately, the user device's limited communication range and battery constraint confirm the unfitness of known robustness metrics suggested for fixed networks, when applied to time-switching communication graphs. Furthermore, the calculation of most of the existing robustness metrics involves non-deterministic polynomial-time complexity, and hence are best-fitted only for small networks. Despite a large volume of works, the complete analysis of a $\textit{low-complexity}$ temporal robustness metric for a communication network is absent in the literature, and the present work aims to fill this gap. More in detail, our work provides a stochastic analysis of network robustness for a massive machine type communication (mMTC) network. The numerical investigation corroborates the exactness of the proposed analytical framework for temporal robustness metric. Along with studying the impact on network robustness of various system parameters, such as cluster head (CH) probability, power threshold value, network size, and node failure probability, we justify the observed trend of numerical results probabilistically.
Improved Pancreatic Tumor Detection by Utilizing Clinically-Relevant Secondary Features
Venue: arXiv cs.CV 2022
Year: 2022

Pancreatic cancer is one of the global leading causes of cancer-related deaths. Despite the success of Deep Learning in computer-aided diagnosis and detection (CAD) methods, little attention has been paid to the detection of Pancreatic Cancer. We propose a method for detecting pancreatic tumor that utilizes clinically-relevant features in the surrounding anatomical structures, thereby better aiming to exploit the radiologist's knowledge compared to other, conventional deep learning approaches. To this end, we collect a new dataset consisting of 99 cases with pancreatic ductal adenocarcinoma (PDAC) and 97 control cases without any pancreatic tumor. Due to the growth pattern of pancreatic cancer, the tumor may not be always visible as a hypodense lesion, therefore experts refer to the visibility of secondary external features that may indicate the presence of the tumor. We propose a method based on a U-Net-like Deep CNN that exploits the following external secondary features: the pancreatic duct, common bile duct and the pancreas, along with a processed CT scan. Using these features, the model segments the pancreatic tumor if it is present. This segmentation for classification and localization approach achieves a performance of 99% sensitivity (one case missed) and 99% specificity, which realizes a 5% increase in sensitivity over the previous state-of-the-art method. The model additionally provides location information with reasonable accuracy and a shorter inference time compared to previous PDAC detection methods. These results offer a significant performance improvement and highlight the importance of incorporating the knowledge of the clinical expert when developing novel CAD methods.
Implementing Window Functions in a Column-Store with Late Materialization (Extended Version)
Venue: arXiv cs.DB 2022
Year: 2022

A window function is a generalization of the aggregation operation. Unlike aggregation, the cardinality of its output is always the same as the cardinality of input. That is, the semantics of this operator imply computing values for extra attributes for each row, depending on its context, either expressed by a sliding window or a previously evaluated row. Window functions are a very powerful tool, which is also popular among data analysts and supported by the majority of industrial DBMSes. It allows to gracefully express quite complex use-cases, such as running sums and averages, local maximum and minimum, and different types of ranking. Since they can be expressed without self-joins and correlated subqueries, their evaluation can be performed much more efficiently. In this paper we discuss an implementation of window functions inside a disk-based column-store with late materialization. Late materialization is a technique that aims to keep tuple reconstruction back from individual columns as long as possible. Initially popular in the late 00's, it is rarely considered nowadays. However, in case of window functions it allows to substantially lower memory footprint. Another contribution of this paper is the application of a segment tree to computing RANGE-based window functions.
Federated Adversarial Learning: A Framework with Convergence Analysis
Venue: arXiv cs.LG 2022
Year: 2022

Federated learning (FL) is a trending training paradigm to utilize decentralized training data. FL allows clients to update model parameters locally for several epochs, then share them to a global model for aggregation. This training paradigm with multi-local step updating before aggregation exposes unique vulnerabilities to adversarial attacks. Adversarial training is a popular and effective method to improve the robustness of networks against adversaries. In this work, we formulate a general form of federated adversarial learning (FAL) that is adapted from adversarial learning in the centralized setting. On the client side of FL training, FAL has an inner loop to generate adversarial samples for adversarial training and an outer loop to update local model parameters. On the server side, FAL aggregates local model updates and broadcast the aggregated model. We design a global robust training loss and formulate FAL training as a min-max optimization problem. Unlike the convergence analysis in classical centralized training that relies on the gradient direction, it is significantly harder to analyze the convergence in FAL for three reasons: 1) the complexity of min-max optimization, 2) model not updating in the gradient direction due to the multi-local updates on the client-side before aggregation and 3) inter-client heterogeneity. We address these challenges by using appropriate gradient approximation and coupling techniques and present the convergence analysis in the over-parameterized regime. Our main result theoretically shows that the minimum loss under our algorithm can converge to $ε$ small with chosen learning rate and communication rounds. It is noteworthy that our analysis is feasible for non-IID clients.
Weakly Supervised Online Action Detection for Infant General Movements
Venue: arXiv cs.CV 2022
Year: 2022

To make the earlier medical intervention of infants' cerebral palsy (CP), early diagnosis of brain damage is critical. Although general movements assessment(GMA) has shown promising results in early CP detection, it is laborious. Most existing works take videos as input to make fidgety movements(FMs) classification for the GMA automation. Those methods require a complete observation of videos and can not localize video frames containing normal FMs. Therefore we propose a novel approach named WO-GMA to perform FMs localization in the weakly supervised online setting. Infant body keypoints are first extracted as the inputs to WO-GMA. Then WO-GMA performs local spatio-temporal extraction followed by two network branches to generate pseudo clip labels and model online actions. With the clip-level pseudo labels, the action modeling branch learns to detect FMs in an online fashion. Experimental results on a dataset with 757 videos of different infants show that WO-GMA can get state-of-the-art video-level classification and cliplevel detection results. Moreover, only the first 20% duration of the video is needed to get classification results as good as fully observed, implying a significantly shortened FMs diagnosis time. Code is available at: https://github.com/scofiedluo/WO-GMA.
A Game-Theoretic Perspective of Generalization in Reinforcement Learning
Venue: arXiv cs.LG 2022
Year: 2022

Generalization in reinforcement learning (RL) is of importance for real deployment of RL algorithms. Various schemes are proposed to address the generalization issues, including transfer learning, multi-task learning and meta learning, as well as the robust and adversarial reinforcement learning. However, there is not a unified formulation of the various schemes, as well as the comprehensive comparisons of methods across different schemes. In this work, we propose a game-theoretic framework for the generalization in reinforcement learning, named GiRL, where an RL agent is trained against an adversary over a set of tasks, where the adversary can manipulate the distributions over tasks within a given threshold. With different configurations, GiRL can reduce the various schemes mentioned above. To solve GiRL, we adapt the widely-used method in game theory, policy space response oracle (PSRO) with the following three important modifications: i) we use model-agnostic meta learning (MAML) as the best-response oracle, ii) we propose a modified projected replicated dynamics, i.e., R-PRD, which ensures the computed meta-strategy of the adversary fall in the threshold, and iii) we also propose a protocol for the few-shot learning of the multiple strategies during testing. Extensive experiments on MuJoCo environments demonstrate that our proposed methods can outperform existing baselines, e.g., MAML.
Vernacular Search Query Translation with Unsupervised Domain Adaptation
Venue: arXiv cs.CL 2022
Year: 2022

With the democratization of e-commerce platforms, an increasingly diversified user base is opting to shop online. To provide a comfortable and reliable shopping experience, it's important to enable users to interact with the platform in the language of their choice. An accurate query translation is essential for Cross-Lingual Information Retrieval (CLIR) with vernacular queries. Due to internet-scale operations, e-commerce platforms get millions of search queries every day. However, creating a parallel training set to train an in-domain translation model is cumbersome. This paper proposes an unsupervised domain adaptation approach to translate search queries without using any parallel corpus. We use an open-domain translation model (trained on public corpus) and adapt it to the query data using only the monolingual queries from two languages. In addition, fine-tuning with a small labeled set further improves the result. For demonstration, we show results for Hindi to English query translation and use mBART-large-50 model as the baseline to improve upon. Experimental results show that, without using any parallel corpus, we obtain more than 20 BLEU points improvement over the baseline while fine-tuning with a small 50k labeled set provides more than 27 BLEU points improvement over the baseline.
Strategic differences between regional investments into graphene technology and how corporations and universities manage patent portfolios
Venue: arXiv cs.DL 2022
Year: 2022

Nowadays, patenting activities are essential in converting applied science to technology in the prevailing innovation model. To gain strategic advantages in the technological competitions between regions, nations need to leverage the investments of public and private funds to diversify over all technologies or specialize in a small number of technologies. In this paper, we investigated who the leaders are at the regional and assignee levels, how they attained their leadership positions, and whether they adopted diversification or specialization strategies, using a dataset of 176,193 patent records on graphene between 1986 and 2017 downloaded from Derwent Innovation. By applying a co-clustering method to the IPC subclasses in the patents and using a z-score method to extract keywords from their titles and abstracts, we identified seven graphene technology areas emerging in the sequence synthesis - composites - sensors - devices - catalyst - batteries - water treatment. We then examined the top regions in their investment preferences and their changes in rankings over time and found that they invested in all seven technology areas. In contrast, at the assignee level, some were diversified while others were specialized. We found that large entities diversified their portfolios across multiple technology areas, while small entities specialized around their core competencies. In addition, we found that universities had higher entropy values than corporations on average, leading us to the hypothesis that corporations file, buy, or sell patents to enable product development. In contrast, universities focus only on licensing their patents. We validated this hypothesis through an aggregate analysis of reassignment and licensing and a more detailed analysis of three case studies - SAMSUNG, RICE UNIVERSITY, and DYSON.
Stochastic Scaling in Loss Functions for Physics-Informed Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

Differential equations are used in a wide variety of disciplines, describing the complex behavior of the physical world. Analytic solutions to these equations are often difficult to solve for, limiting our current ability to solve complex differential equations and necessitating sophisticated numerical methods to approximate solutions. Trained neural networks act as universal function approximators, able to numerically solve differential equations in a novel way. In this work, methods and applications of neural network algorithms for numerically solving differential equations are explored, with an emphasis on varying loss functions and biological applications. Variations on traditional loss function and training parameters show promise in making neural network-aided solutions more efficient, allowing for the investigation of more complex equations governing biological principles.
Comparison Conundrum and the Chamber of Visualizations: An Exploration of How Language Influences Visual Design
Venue: arXiv cs.HC 2022
Year: 2022

The language for expressing comparisons is often complex and nuanced, making supporting natural language-based visual comparison a non-trivial task. To better understand how people reason about comparisons in natural language, we explore a design space of utterances for comparing data entities. We identified different parameters of comparison utterances that indicate what is being compared (i.e., data variables and attributes) as well as how these parameters are specified (i.e., explicitly or implicitly). We conducted a user study with sixteen data visualization experts and non-experts to investigate how they designed visualizations for comparisons in our design space. Based on the rich set of visualization techniques observed, we extracted key design features from the visualizations and synthesized them into a subset of sixteen representative visualization designs. We then conducted a follow-up study to validate user preferences for the sixteen representative visualizations corresponding to utterances in our design space. Findings from these studies suggest guidelines and future directions for designing natural language interfaces and recommendation tools to better support natural language comparisons in visual analytics.
Global Hierarchical Attention for 3D Point Cloud Analysis
Venue: arXiv cs.CV 2022
Year: 2022

We propose a new attention mechanism, called Global Hierarchical Attention (GHA), for 3D point cloud analysis. GHA approximates the regular global dot-product attention via a series of coarsening and interpolation operations over multiple hierarchy levels. The advantage of GHA is two-fold. First, it has linear complexity with respect to the number of points, enabling the processing of large point clouds. Second, GHA inherently possesses the inductive bias to focus on spatially close points, while retaining the global connectivity among all points. Combined with a feedforward network, GHA can be inserted into many existing network architectures. We experiment with multiple baseline networks and show that adding GHA consistently improves performance across different tasks and datasets. For the task of semantic segmentation, GHA gives a +1.7% mIoU increase to the MinkowskiEngine baseline on ScanNet. For the 3D object detection task, GHA improves the CenterPoint baseline by +0.5% mAP on the nuScenes dataset, and the 3DETR baseline by +2.1% mAP25 and +1.5% mAP50 on ScanNet.
Automatic Security Assessment of GitHub Actions Workflows
Venue: arXiv cs.CR 2022
Year: 2022

The demand for quick and reliable DevOps operations pushed distributors of repository platforms to implement workflows. Workflows allow automating code management operations directly on the repository hosting the software. However, this feature also introduces security issues that directly affect the repository, its content, and all the software supply chains in which the hosted code is involved in. Hence, an attack exploiting vulnerable workflows can affect disruptively large software ecosystems. To empirically assess the importance of this problem, in this paper, we focus on the de-facto main distributor (i.e., GitHub), and we developed a security assessment methodology for GitHub Actions workflows, which are widely adopted in software supply chains. We implemented the methodology in a tool (GHAST) and applied it on 50 open-source projects. The experimental results are worrisome as they allowed identifying a total of 24,905 security issues (all reported to the corresponding stakeholders), thereby indicating that the problem is open and demands further research and investigation.