OpenRaiser/Intern-Atlas huggingface dataset:
CSSAM:Code Search via Attention Matching of Code Semantics and Structures
Venue: arXiv cs.SE 2022
Year: 2022

Despite the continuous efforts in improving both the effectiveness and efficiency of code search, two issues remained unsolved. First, programming languages have inherent strong structural linkages, and feature mining of code as text form would omit the structural information contained inside it. Second, there is a potential semantic relationship between code and query, it is challenging to align code and text across sequences so that vectors are spatially consistent during similarity matching. To tackle both issues, in this paper, a code search model named CSSAM (Code Semantics and Structures Attention Matching) is proposed. By introducing semantic and structural matching mechanisms, CSSAM effectively extracts and fuses multidimensional code features. Specifically, the cross and residual layer was developed to facilitate high-latitude spatial alignment of code and query at the token level. By leveraging the residual interaction, a matching module is designed to preserve more code semantics and descriptive features, that enhances the adhesion between the code and its corresponding query text. Besides, to improve the model's comprehension of the code's inherent structure, a code representation structure named CSRG (Code Semantic Representation Graph) is proposed for jointly representing abstract syntax tree nodes and the data flow of the codes. According to the experimental results on two publicly available datasets containing 540k and 330k code segments, CSSAM significantly outperforms the baselines in terms of achieving the highest SR@1/5/10, MRR, and NDCG@50 on both datasets respectively. Moreover, the ablation study is conducted to quantitatively measure the impact of each key component of CSSAM on the efficiency and effectiveness of code search, which offers the insights into the improvement of advanced code search solutions.
AWEncoder: Adversarial Watermarking Pre-trained Encoders in Contrastive Learning
Venue: arXiv cs.CV 2022
Year: 2022

As a self-supervised learning paradigm, contrastive learning has been widely used to pre-train a powerful encoder as an effective feature extractor for various downstream tasks. This process requires numerous unlabeled training data and computational resources, which makes the pre-trained encoder become valuable intellectual property of the owner. However, the lack of a priori knowledge of downstream tasks makes it non-trivial to protect the intellectual property of the pre-trained encoder by applying conventional watermarking methods. To deal with this problem, in this paper, we introduce AWEncoder, an adversarial method for watermarking the pre-trained encoder in contrastive learning. First, as an adversarial perturbation, the watermark is generated by enforcing the training samples to be marked to deviate respective location and surround a randomly selected key image in the embedding space. Then, the watermark is embedded into the pre-trained encoder by further optimizing a joint loss function. As a result, the watermarked encoder not only performs very well for downstream tasks, but also enables us to verify its ownership by analyzing the discrepancy of output provided using the encoder as the backbone under both white-box and black-box conditions. Extensive experiments demonstrate that the proposed work enjoys pretty good effectiveness and robustness on different contrastive learning algorithms and downstream tasks, which has verified the superiority and applicability of the proposed work.
Extrinsic Camera Calibration with Semantic Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

Monocular camera sensors are vital to intelligent vehicle operation and automated driving assistance and are also heavily employed in traffic control infrastructure. Calibrating the monocular camera, though, is time-consuming and often requires significant manual intervention. In this work, we present an extrinsic camera calibration approach that automatizes the parameter estimation by utilizing semantic segmentation information from images and point clouds. Our approach relies on a coarse initial measurement of the camera pose and builds on lidar sensors mounted on a vehicle with high-precision localization to capture a point cloud of the camera environment. Afterward, a mapping between the camera and world coordinate spaces is obtained by performing a lidar-to-camera registration of the semantically segmented sensor data. We evaluate our method on simulated and real-world data to demonstrate low error measurements in the calibration results. Our approach is suitable for infrastructure sensors as well as vehicle sensors, while it does not require motion of the camera platform.
Abutting Grating Illusion: Cognitive Challenge to Neural Network Models
Venue: arXiv cs.CV 2022
Year: 2022

Even the state-of-the-art deep learning models lack fundamental abilities compared to humans. Multiple comparison paradigms have been proposed to explore the distinctions between humans and deep learning. While most comparisons rely on corruptions inspired by mathematical transformations, very few have bases on human cognitive phenomena. In this study, we propose a novel corruption method based on the abutting grating illusion, which is a visual phenomenon widely discovered in both human and a wide range of animal species. The corruption method destroys the gradient-defined boundaries and generates the perception of illusory contours using line gratings abutting each other. We applied the method on MNIST, high resolution MNIST, and silhouette object images. Various deep learning models are tested on the corruption, including models trained from scratch and 109 models pretrained with ImageNet or various data augmentation techniques. Our results show that abutting grating corruption is challenging even for state-of-the-art deep learning models because most models are randomly guessing. We also discovered that the DeepAugment technique can greatly improve robustness against abutting grating illusion. Visualisation of early layers indicates that better performing models exhibit stronger end-stopping property, which is consistent with neuroscience discoveries. To validate the corruption method, 24 human subjects are involved to classify samples of corrupted datasets.
One-Hop Listening Based ARQs for Low-Latency Communication in Multi-Hop Networks
Venue: arXiv cs.IT 2022
Year: 2022

Inspired by emerging applications in vehicular networks, we address the problem of achieving high-reliability and low-latency communication in multi-hop wireless networks. We propose a new family of Automatic-Repeat-Requests (ARQs) based cooperative strategies wherein high end-to-end reliability is obtained using packet re-transmissions at each hop while the low-latency constraint is met by imposing an upper bound on the total number of packet retransmissions across the network. A hallmark of our strategies is the one-hop listening capability wherein nodes utilize the unused ARQs of their preceding node just by counting the number of failed attempts due to decoding errors. We further extend the idea of one-hop listening to multi-hop listening, wherein a set of consecutive nodes form clusters to utilize the unused ARQs of the preceding nodes, beyond its nearest neighbour, to further improve reliability. Thus, our strategies provide the high-reliability feature with no compromise in the original latency-constraint. For the proposed strategies, we solve non-linear optimization problems on distributing the ARQs across the nodes so as to minimize packet drop probability (PDP) subject to a total number of ARQs in the network. Through extensive theoretical results on PDP and delay profiles, we show that the proposed strategies outperform the best-known strategies in this space.
CaST: A Toolchain for Creating and Characterizing Realistic Wireless Network Emulation Scenarios
Venue: arXiv cs.NI 2022
Year: 2022

Large-scale wireless testbeds are being increasingly used in developing and evaluating new solutions for next generation wireless networks. Among others, high-fidelity FPGA-based emulation platforms have unique capabilities for faithfully modeling real-world wireless environments in real-time and at scale, while guaranteeing repeatability. However, the reliability of the solutions tested on emulation platforms heavily depends on the precision of the emulation process, which is often overlooked. To address this unmet need in wireless network emulator-based experiments, in this paper we present CaST, a Channel emulation generator and Sounder Toolchain for creating and characterizing realistic wireless network scenarios with high accuracy. CaST consists of (i) a framework for creating mobile wireless scenarios from ray-tracing models for FPGA-based emulation platforms, and (ii) a containerized Software Defined Radio-based channel sounder to precisely characterize the emulated channels. We demonstrate the use of CaST by designing, deploying and validating multi-path mobile scenarios on Colosseum, the world's largest wireless network emulator. Results show that CaST achieves <= 20 ns accuracy in sounding Channel Impulse Response tap delays, and 0.5 dB accuracy in measuring tap gains.
I Learn to Diffuse, or Data Alchemy 101: a Mnemonic Manifesto
Venue: arXiv cs.HC 2022
Year: 2022

In this manifesto, we put forward the idea of data alchemy as a narrative device to discuss storytelling and transdisciplinarity in visualization. If data is the prima materia of modern science, how does one perform the Great Work? We use text-to-image diffusion-based generative art to develop the concept, and structure our argument in ten propositions, as if they were ten issues of a comic novel on data alchemy: Ad Disco Diffusionem. To follow the argument, the reader must immerse themselves in our miro board, and navigate a multimedia semiotic topology that includes comics, videos, code demos, and ergotic literature in a true alchemic sense. By accessing this paradigm one might find new sources of inspiration for scientific inquiry in familiar places, or get lost in the creative exploration of the unknown. Our colorful, sometimes poetic, exposition should not distract the reader from the seriousness of the ideas discussed, but ultimately it is about the journey.
Application of Guessing to Sequential Decoding of Polarization-Adjusted Convolutional (PAC) Codes
Venue: arXiv cs.IT 2022
Year: 2022

Despite the extreme error-correction performance, the amount of computation of sequential decoding of the polarization-adjusted convolutional (PAC) codes is random. In sequential decoding of convolutional codes, the computational cutoff rate denotes the region between rates whose average computational complexity of decoding is finite and those which is infinite. In this paper, by benefiting from the polarization and guessing techniques, we prove that the computational cutoff rate in sequential decoding of pre-transformed polar codes polarizes. The polarization of the computational cutoff rate affects the criteria for the rate-profile construction of the pre-transformed polar codes. We propose a technique for taming the Reed-Muller (RM) rate-profile construction, and the performance results demonstrate that the error-correction performance of the PAC codes can achieve the theoretical bounds using the tamed-RM rate-profile construction and requires a significantly lower computational complexity than the RM rate-profile construction.
Eight Years of Face Recognition Research: Reproducibility, Achievements and Open Issues
Venue: arXiv cs.CV 2022
Year: 2022

Automatic face recognition is a research area with high popularity. Many different face recognition algorithms have been proposed in the last thirty years of intensive research in the field. With the popularity of deep learning and its capability to solve a huge variety of different problems, face recognition researchers have concentrated effort on creating better models under this paradigm. From the year 2015, state-of-the-art face recognition has been rooted in deep learning models. Despite the availability of large-scale and diverse datasets for evaluating the performance of face recognition algorithms, many of the modern datasets just combine different factors that influence face recognition, such as face pose, occlusion, illumination, facial expression and image quality. When algorithms produce errors on these datasets, it is not clear which of the factors has caused this error and, hence, there is no guidance in which direction more research is required. This work is a followup from our previous works developed in 2014 and eventually published in 2016, showing the impact of various facial aspects on face recognition algorithms. By comparing the current state-of-the-art with the best systems from the past, we demonstrate that faces under strong occlusions, some types of illumination, and strong expressions are problems mastered by deep learning algorithms, whereas recognition with low-resolution images, extreme pose variations, and open-set recognition is still an open problem. To show this, we run a sequence of experiments using six different datasets and five different face recognition algorithms in an open-source and reproducible manner. We provide the source code to run all of our experiments, which is easily extensible so that utilizing your own deep network in our evaluation is just a few minutes away.
Affective Learning Objectives for Communicative Visualizations
Venue: arXiv cs.HC 2022
Year: 2022

When designing communicative visualizations, we often focus on goals that seek to convey patterns, relations, or comparisons (cognitive learning objectives). We pay less attention to affective intents--those that seek to influence or leverage the audience's opinions, attitudes, or values in some way. Affective objectives may range in outcomes from making the viewer care about the subject, strengthening a stance on an opinion, or leading them to take further action. Because such goals are often considered a violation of perceived 'neutrality' or are 'political,' designers may resist or be unable to describe these intents, let alone formalize them as learning objectives. While there are notable exceptions--such as advocacy visualizations or persuasive cartography--we find that visualization designers rarely acknowledge or formalize affective objectives. Through interviews with visualization designers, we expand on prior work on using learning objectives as a framework for describing and assessing communicative intent. Specifically, we extend and revise the framework to include a set of affective learning objectives. This structured taxonomy can help designers identify and declare their goals and compare and assess designs in a more principled way. Additionally, the taxonomy can enable external critique and analysis of visualizations. We illustrate the use of the taxonomy with a critical analysis of an affective visualization.
Practitioners Versus Users: A Value-Sensitive Evaluation of Current Industrial Recommender System Design
Venue: arXiv cs.CY 2022
Year: 2022

Recommender systems are playing an increasingly important role in alleviating information overload and supporting users' various needs, e.g., consumption, socialization, and entertainment. However, limited research focuses on how values should be extensively considered in industrial deployments of recommender systems, the ignorance of which can be problematic. To fill this gap, in this paper, we adopt Value Sensitive Design to comprehensively explore how practitioners and users recognize different values of current industrial recommender systems. Based on conceptual and empirical investigations, we focus on five values: recommendation quality, privacy, transparency, fairness, and trustworthiness. We further conduct in-depth qualitative interviews with 20 users and 10 practitioners to delve into their opinions about these values. Our results reveal the existence and sources of tensions between practitioners and users in terms of value interpretation, evaluation, and practice, which provide novel implications for designing more human-centric and value-sensitive recommender systems.
Adversarial Attacks on Image Generation With Made-Up Words
Venue: arXiv cs.CV 2022
Year: 2022

Text-guided image generation models can be prompted to generate images using nonce words adversarially designed to robustly evoke specific visual concepts. Two approaches for such generation are introduced: macaronic prompting, which involves designing cryptic hybrid words by concatenating subword units from different languages; and evocative prompting, which involves designing nonce words whose broad morphological features are similar enough to that of existing words to trigger robust visual associations. The two methods can also be combined to generate images associated with more specific visual concepts. The implications of these techniques for the circumvention of existing approaches to content moderation, and particularly the generation of offensive or harmful images, are discussed.
DALLE-URBAN: Capturing the urban design expertise of large text to image transformers
Venue: arXiv cs.CV 2022
Year: 2022

Automatically converting text descriptions into images using transformer architectures has recently received considerable attention. Such advances have implications for many applied design disciplines across fashion, art, architecture, urban planning, landscape design and the future tools available to such disciplines. However, a detailed analysis capturing the capabilities of such models, specifically with a focus on the built environment, has not been performed to date. In this work, we investigate the capabilities and biases of such text-to-image methods as it applies to the built environment in detail. We use a systematic grammar to generate queries related to the built environment and evaluate resulting generated images. We generate 1020 different images and find that text to image transformers are robust at generating realistic images across different domains for this use-case. Generated imagery can be found at the github: https://github.com/sachith500/DALLEURBAN
A Historical Interaction between Artificial Intelligence and Philosophy
Venue: arXiv cs.AI 2022
Year: 2022

This paper reviews the historical development of AI and representative philosophical thinking from the perspective of the research paradigm. Additionally, it considers the methodology and applications of AI from a philosophical perspective and anticipates its continued advancement. In the history of AI, Symbolism and connectionism are the two main paradigms in AI research. Symbolism holds that the world can be explained by symbols and dealt with through precise, logical processes, but connectionism believes this process should be implemented through artificial neural networks. Regardless of how intelligent machines or programs should achieve their smart goals, the historical development of AI demonstrates the best answer at this time. Still, it is not the final answer of AI research.
Vision-Based Activity Recognition in Children with Autism-Related Behaviors
Venue: arXiv cs.CV 2022
Year: 2022

Advances in machine learning and contactless sensors have enabled the understanding complex human behaviors in a healthcare setting. In particular, several deep learning systems have been introduced to enable comprehensive analysis of neuro-developmental conditions such as Autism Spectrum Disorder (ASD). This condition affects children from their early developmental stages onwards, and diagnosis relies entirely on observing the child's behavior and detecting behavioral cues. However, the diagnosis process is time-consuming as it requires long-term behavior observation, and the scarce availability of specialists. We demonstrate the effect of a region-based computer vision system to help clinicians and parents analyze a child's behavior. For this purpose, we adopt and enhance a dataset for analyzing autism-related actions using videos of children captured in uncontrolled environments (e.g. videos collected with consumer-grade cameras, in varied environments). The data is pre-processed by detecting the target child in the video to reduce the impact of background noise. Motivated by the effectiveness of temporal convolutional models, we propose both light-weight and conventional models capable of extracting action features from video frames and classifying autism-related behaviors by analyzing the relationships between frames in a video. Through extensive evaluations on the feature extraction and learning strategies, we demonstrate that the best performance is achieved with an Inflated 3D Convnet and Multi-Stage Temporal Convolutional Networks, achieving a 0.83 Weighted F1-score for classification of the three autism-related actions, outperforming existing methods. We also propose a light-weight solution by employing the ESNet backbone within the same system, achieving competitive results of 0.71 Weighted F1-score, and enabling potential deployment on embedded systems.
Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval
Venue: arXiv cs.IR 2022
Year: 2022

In this paper, we propose a new dense retrieval model which learns diverse document representations with deep query interactions. Our model encodes each document with a set of generated pseudo-queries to get query-informed, multi-view document representations. It not only enjoys high inference efficiency like the vanilla dual-encoder models, but also enables deep query-document interactions in document encoding and provides multi-faceted representations to better match different queries. Experiments on several benchmarks demonstrate the effectiveness of the proposed method, out-performing strong dual encoder baselines.The code is available at \url{https://github.com/jordane95/dual-cross-encoder
Sampling, Communication, and Prediction Co-Design for Synchronizing the Real-World Device and Digital Model in Metaverse
Venue: arXiv cs.RO 2022
Year: 2022

The metaverse has the potential to revolutionize the next generation of the Internet by supporting highly interactive services with the help of Mixed Reality (MR) technologies; still, to provide a satisfactory experience for users, the synchronization between the physical world and its digital models is crucial. This work proposes a sampling, communication and prediction co-design framework to minimize the communication load subject to a constraint on tracking the Mean Squared Error (MSE) between a real-world device and its digital model in the metaverse. To optimize the sampling rate and the prediction horizon, we exploit expert knowledge and develop a constrained Deep Reinforcement Learning (DRL) algorithm, named Knowledge-assisted Constrained Twin-Delayed Deep Deterministic (KC-TD3) policy gradient algorithm. We validate our framework on a prototype composed of a real-world robotic arm and its digital model. Compared with existing approaches: (1) When the tracking error constraint is stringent (MSE=0.002 degrees), our policy degenerates into the policy in the sampling-communication co-design framework. (2) When the tracking error constraint is mild (MSE=0.007 degrees), our policy degenerates into the policy in the prediction-communication co-design framework. (3) Our framework achieves a better trade-off between the average MSE and the average communication load compared with a communication system without sampling and prediction. For example, the average communication load can be reduced up to 87% when the track error constraint is 0.002 degrees. (4) Our policy outperforms the benchmark with the static sampling rate and prediction horizon optimized by exhaustive search, in terms of the tail probability of the tracking error. Furthermore, with the assistance of expert knowledge, the proposed algorithm KC-TD3 achieves better convergence time, stability, and final policy performance.