Reviewerly/exHarmony huggingface dataset:
Patching Textiles

Textiles have several characteristics that make them well suited for updates, sometimes called patching or mending, but textile repair is underexplored in the context of personal fabrication. This exploration is an urgent sustainability issue so we can extend the life of textiles and avoid producing more materials. In this paper we take a craft ethnography approach by interviewing 15 visible mending educators for insights into how they teach the techniques of repair and re-use so individuals can upcycle the textiles they already own. We discuss the values that menders bring to the practice, the teaching strategies they employ, the tangible teaching materials and tools of the practice, and introduce three types of teaching samplers: wearable samplers, sampler swatches, and practice samplers. Overall, these interviews provide insights for textile maker toolkits, textile personal fabrication, and how we can teach tangible hybrid crafts and sustainable making practices.
Leveraging Meta-path Contexts for Classification in Heterogeneous Information Networks

A heterogeneous information network (HIN) has as vertices objects of different types and as edges the relations between objects, which are also of various types. We study the problem of classifying objects in HINs. Most existing methods perform poorly when given scarce labeled objects as training sets, and methods that improve classification accuracy under such scenarios are often computationally expensive. To address these problems, we propose ConCH, a graph neural network model. ConCH formulates the classification problem as a multitask learning problem that combines semi-supervised learning with self-supervised learning to learn from both labeled and unlabeled data. ConCH employs meta-paths, which are sequences of object types that capture semantic relationships between objects. ConCH co-derives object embeddings and context embeddings via graph convolution. It also uses the attention mechanism to fuse such embeddings. We conduct extensive experiments to evaluate the performance of ConCH against other 15 classification methods. Our results show that ConCH is an effective and efficient method for HIN classification.
Multi-Modal Temporal Convolutional Network for Anticipating Actions in Egocentric Videos

Anticipating human actions is an important task that needs to be addressed for the development of reliable intelligent agents, such as self-driving cars or robot assistants. While the ability to make future predictions with high accuracy is crucial for designing the anticipation approaches, the speed at which the inference is performed is not less important. Methods that are accurate but not sufficiently fast would introduce a high latency into the decision process. Thus, this will increase the reaction time of the underlying system. This poses a problem for domains such as autonomous driving, where the reaction time is crucial. In this work, we propose a simple and effective multi-modal architecture based on temporal convolutions. Our approach stacks a hierarchy of temporal convolutional layers and does not rely on recurrent layers to ensure a fast prediction. We further introduce a multi-modal fusion mechanism that captures the pairwise interactions between RGB, flow, and object modalities. Results on two large-scale datasets of egocentric videos, EPIC-Kitchens-55 and EPIC-Kitchens-100, show that our approach achieves comparable performance to the state-of-the-art approaches while being significantly faster.
Architectural Adversarial Robustness: The Case for Deep Pursuit

Despite their unmatched performance, deep neural networks remain susceptible to targeted attacks by nearly imperceptible levels of adversarial noise. While the underlying cause of this sensitivity is not well understood, theoretical analyses can be simplified by reframing each layer of a feed-forward network as an approximate solution to a sparse coding problem. Iterative solutions using basis pursuit are theoretically more stable and have improved adversarial robustness. However, cascading layer-wise pursuit implementations suffer from error accumulation in deeper networks. In contrast, our new method of deep pursuit approximates the activations of all layers as a single global optimization problem, allowing us to consider deeper, real-world architectures with skip connections such as residual networks. Experimentally, our approach demonstrates improved robustness to adversarial noise.
Robust Vehicle Re-identification via Rigid Structure Prior

Vehicle re-identification (re-id) is one of the most important components in the current intelligence transport system, benefiting both the smart traffic management and the optimal path planning. In this paper, we focus on developing a robust part-aware structure-based vehicle re-id system against the massive appearance changes due to the pose and illumination variants. Specifically, we apply the strong convolutional neural networks to extract the visual representation, which is based on the detected vehicle images. Taking one step further, we deploy a part detector to recognize different vehicle parts, such as front, back, left, and right, which explicitly introduce the prior knowledge on the structure of the rigid objective, i.e., vehicle. With the geometry information, we further harness different part feature extractors to filter wrong matches. By using this simple but effective strategy, we remove the hard negative candidates while maintaining high recall accuracy, combing general global-level coarse-grained re-id feature models with part-level fine-grained features. We achieved 71.51% mAP in the vehicle re-id track of the AI City Challenge 2021, which verified the effectiveness and scalability of the proposed structure-based method. The code will be available at https://github.com/XuanmengZhang/AICITY2021-Track2.
Decoding Word Embeddings with Brain-Based Semantic Features

Word embeddings are vectorial semantic representations built with either counting or predicting techniques aimed at capturing shades of meaning from word co-occurrences. Since their introduction, these representations have been criticized for lacking interpretable dimensions. This property of word embeddings limits our understanding of the semantic features they actually encode. Moreover, it contributes to the “black box” nature of the tasks in which they are used, since the reasons for word embedding performance often remain opaque to humans. In this contribution, we explore the semantic properties encoded in word embeddings by mapping them onto interpretable vectors, consisting of explicit and neurobiologically motivated semantic features (Binder et al. 2016). Our exploration takes into account different types of embeddings, including factorized count vectors and predict models (Skip-Gram, GloVe, etc.), as well as the most recent contextualized representations (i.e., ELMo and BERT). In our analysis, we first evaluate the quality of the mapping in a retrieval task, then we shed light on the semantic features that are better encoded in each embedding type. A large number of probing tasks is finally set to assess how the original and the mapped embeddings perform in discriminating semantic categories. For each probing task, we identify the most relevant semantic features and we show that there is a correlation between the embedding performance and how they encode those features. This study sets itself as a step forward in understanding which aspects of meaning are captured by vector spaces, by proposing a new and simple method to carve human-interpretable semantic representations from distributional vectors.
API2Com: On the Improvement of Automatically Generated Code Comments Using API Documentations

Code comments can help in program comprehension and are considered as important artifacts to help developers in software maintenance. However, the comments are mostly missing or are outdated, specially in complex software projects. As a result, several automatic comment generation models are developed as a solution. The recent models explore the integration of external knowledge resources such as Unified Modeling Language class diagrams to improve the generated comments. In this paper, we propose API2Com, a model that leverages the Application Programming Interface Documentations (API Docs) as a knowledge resource for comment generation. The API Docs include the description of the methods in more details and therefore, can provide better context in the generated comments. The API Docs are used along with the code snippets and Abstract Syntax Trees in our model.We apply the model on a large Java dataset of over 130,000 methods and evaluate it using both Transformer and RNN- base architectures. Interestingly, when API Docs are used, the performance increase is negligible. We therefore run different experiments to reason about the results. For methods that only contain one API, adding API Docs improves the results by 4% BLEU score on average (BLEU score is an automatic evaluation metric used in machine translation). However, as the number of APIs that are used in a method increases, the performance of the model in generating comments decreases due to long documentations used in the input. Our results confirm that the API Docs can be useful in generating better comments, but, new techniques are required to identify the most informative ones in a method rather than using all documentations simultaneously.
A novel periodic learning ontology matching model based on interactive grasshopper optimization algorithm

The ontology matching is a significant task for data integration and semantic interoperability. Although a large number of effective ontology matching methods have been proposed in a fully automated way, user involvement during the matching process is needed for real-world applications. It has been recognized as an effective method for further improving the quality of matching, especially for very precise matching cases. However, involving users during complex matching process suffers from new challenges of how to reduce the burden on users and how to increase effective interaction. In this paper, we propose a novel periodic learning ontology matching model based on interactive grasshopper optimization algorithm to address the above-mentioned issues. This new model takes into account the periodic feedback from users during the optimization process, rather than every generation, and a roulette wheel method is introduced to select the most problematic candidate mappings to present to users, not all, and to reduce the burden on users. To ensure the effectiveness of the interaction, a reward and punishment mechanism is considered for candidate mappings to propagate the feedback of user, and to guide the search direction of the algorithm. The experiments, conducted on two interactive tracks from Ontology Alignment Evaluation Initiative (OAEI), show that the proposed model significantly improve the quality of matching. Compared to other state-of-the-art matching systems, our model outperforms other methods in almost all cases with given different error rate, which makes it one of the most advanced leaders. Finally, a typical case of data integration is studied to present how the proposed approach is able to help enterprises to harmonize product catalogs.
Point cloud denoising using non-local collaborative projections

Point cloud is important for object detection and recognition. The main challenge of point cloud denoising is to preserve the geometric structures. Several state-of-the-art point cloud denoising methods focus only on analyzing local geometric information, which is sensitive to noise and outliers. In this paper, we propose a novel point cloud denoising algorithm based on the characteristics of non-local self-similarity. First, we present an adaptive curvature threshold to select the edge points and tune their corresponding normals, which can preserve the sharp details. Then we propose a structure-aware descriptor called projective height vector to capture the local height variations by normal height projection and the most similar non-local projective height vectors are grouped into a height matrix to enhance the structure representation. Moreover, the proposed structure descriptor is invariant with rigid transformation. Finally, an improved weighted nuclear norm minimization is proposed to optimize the height matrix and reconstruct a high-quality point cloud. Rather than treating each singular value independently, each component in our proposed weight definition connects with the most important components to preserve the major structural information. Experiments on synthetic and scanned point cloud datasets demonstrate that our algorithm outperforms state-of-the-art methods in terms of reconstruction accuracy and structure preservation.
Image Compression with Recurrent Neural Network and Generalized Divisive Normalization

Image compression is a method to remove spatial redundancy between adjacent pixels and reconstruct a high-quality image. In the past few years, deep learning has gained huge attention from the research community and produced promising image reconstruction results. Therefore, recent methods focused on developing deeper and more complex networks, which significantly increased network complexity. In this paper, two effective novel blocks are developed: analysis and synthesis block that employs the convolution layer and Generalized Divisive Normalization (GDN) in the variable-rate encoder and decoder side. Our network utilizes a pixel RNN approach for quantization. Furthermore, to improve the whole network, we encode a residual image using LSTM cells to reduce unnecessary information. Experimental results demonstrated that the pro-posed variable-rate framework with novel blocks outperforms existing methods and standard image codecs, such as George’s [11] and JPEG in terms of image similarity. The project page along with code and models are available at https://github.com/khawar512/cvpr_image_compress
Adversarial Imaging Pipelines

Adversarial attacks play a critical role in understanding deep neural network predictions and improving their robustness. Existing attack methods aim to deceive convolutional neural network (CNN)-based classifiers by manipulating RGB images that are fed directly to the classifiers. However, these approaches typically neglect the influence of the camera optics and image processing pipeline (ISP) that produce the network inputs. ISPs transform RAW measurements to RGB images and traditionally are assumed to preserve adversarial patterns. In fact, these low-level pipelines can destroy, introduce or amplify adversarial patterns that can deceive a downstream detector. As a result, optimized patterns can become adversarial for the classifier after being transformed by a certain camera ISP or optical lens system but not for others. In this work, we examine and develop such an attack that deceives a specific camera ISP while leaving others intact, using the same downstream classifier. We frame this camera-specific attack as a multi-task optimization problem, relying on a differentiable approximation for the ISP itself. We validate the proposed method using recent state-of-the-art automotive hardware ISPs, achieving 92% fooling rate when attacking a specific ISP. We demonstrate physical optics attacks with 90% fooling rate for a specific camera lens.
Importance of different facial parts for face detection networks

Most state-of-the-art face detection algorithms are usually trained with full-face pictures, without any occlusions. The first novel contribution of this paper is an analysis of the accuracy of three off-the-shelf face detection algorithms (MTCNN, Retinaface, and DLIB) on occluded faces. In order to determine the importance of different facial parts, the face detection accuracy is evaluated in two settings: Firstly, we automatically modify the CFP dataset and remove different areas of each face: We overlay a grid over each face and remove one cell at a time. Similarly, we overlay a rectangle over the main landmarks of a face - eye(s), nose and mouth. Furthermore, we resemble a face mask by overlaying a rectangle starting from the bottom of the face. Secondly, we test the performance of the algorithms on people with real-world face masks. The second contribution of this paper is the discovery of a previously unknown behaviour of the widely used MTCNN face detection algorithm-if there is a face inside another face, MTCNN does not detect the larger face.
A Large-scale Study of Security Vulnerability Support on Developer Q&amp;A Websites

Context: Developers usually seek solutions to addressing Security Vulnerabilities (SVs) on developer Question and Answer (Q&A) websites. However, there is still little known about these SV-specific discussions on different Q&A sites. Objective: We present a large-scale empirical study to understand developers’ SV discussions and how these discussions are being supported by Q&A sites. Method: We use topic modeling to uncover the topics of 71,329 curated SV posts from two large Q&A sites, namely Stack Overflow (SO) and Security StackExchange (SSE). We then analyze the popularity, difficulty, and level of expertise for each topic. We also perform a qualitative analysis to identify the types of solutions to SV-related questions. Results: We identify 13 main SV discussion topics. Many topics do not follow the distributions and trends in expert-based security sources, e.g., Common Weakness Enumeration (CWE) and Open Web Application Security Project (OWASP). We also discover that SV discussions attract more experts to answer than many other domains, but some difficult SV topics (e.g., Vulnerability Scanning Tools) still receive quite limited support from experts. Moreover, we identify seven key types of answers given to SV questions, in which SO often provides code and instructions, while SSE usually gives experience-based advice and explanations. Conclusion: Our findings provide support for researchers and practitioners to effectively acquire, share and leverage SV knowledge on Q&A sites.
Toward Region-Aware Attention Learning for Scene Graph Generation

Scene graph generation (SGGen) is a challenging task due to a complex visual context of an image. Intuitively, the human visual system can volitionally focus on attended regions by salient stimuli associated with visual cues. For example, to infer the relationship between man and horse, the interaction between human leg and horseback can provide strong visual evidence to predict the predicate ride. Besides, the attended region face can also help to determine the object man. Till now, most of the existing works studied the SGGen by extracting coarse-grained bounding box features while understanding fine-grained visual regions received limited attention. To mitigate the drawback, this article proposes a region-aware attention learning method. The key idea is to explicitly construct the attention space to explore salient regions with the object and predicate inferences. First, we extract a set of regions in an image with the standard detection pipeline. Each region regresses to an object. Second, we propose the object-wise attention graph neural network (GNN), which incorporates attention modules into the graph structure to discover attended regions for object inference. Third, we build the predicate-wise co-attention GNN to jointly highlight subject's and object's attended regions for predicate inference. Particularly, each subject-object pair is connected with one of the latent predicates to construct one triplet. The proposed intra-triplet and inter-triplet learning mechanism can help discover the pair-wise attended regions to infer predicates. Extensive experiments on two popular benchmarks demonstrate the superiority of the proposed method. Additional ablation studies and visualization further validate its effectiveness.
Kamino

Organizations are increasingly relying on data to support decisions. When data contains private and sensitive information, the data owner often desires to publish a synthetic database instance that is similarly useful as the true data, while ensuring the privacy of individual data records. Existing differentially private data synthesis methods aim to generate useful data based on applications, but they fail in keeping one of the most fundamental data properties of the structured data --- the underlying correlations and dependencies among tuples and attributes (i.e., the structure of the data). This structure is often expressed as integrity and schema constraints, or with a probabilistic generative process. As a result, the synthesized data is not useful for any downstream tasks that require this structure to be preserved. This work presents KAMINO, a data synthesis system to ensure differential privacy and to preserve the structure and correlations present in the original dataset. KAMINO takes as input of a database instance, along with its schema (including integrity constraints), and produces a synthetic database instance with differential privacy and structure preservation guarantees. We empirically show that while preserving the structure of the data, KAMINO achieves comparable and even better usefulness in applications of training classification models and answering marginal queries than the state-of-the-art methods of differentially private data synthesis.
Transitional Adaptation of Pretrained Models for Visual Storytelling

Previous models for vision-to-language generation tasks usually pretrain a visual encoder and a language generator in the respective domains and jointly finetune them with the target task. However, this direct transfer practice may suffer from the discord between visual specificity and language fluency since they are often separately trained from large corpora of visual and text data with no common ground. In this work, we claim that a transitional adaptation task is required between pretraining and finetuning to harmonize the visual encoder and the language model for challenging downstream target tasks like visual storytelling. We propose a novel approach named Transitional Adaptation of Pre-trained Model (TAPM) that adapts the multi-modal modules to each other with a simpler alignment task between visual inputs only with no need for text labels. Through extensive experiments, we show that the adaptation step significantly improves the performance of multiple language models for sequential video and image captioning tasks. We achieve new state-of-the-art performance on both language metrics and human evaluation in the multi-sentence description task of LSMDC 2019 [50] and the image storytelling task of VIST [18]. Our experiments reveal that this improvement in caption quality does not depend on the specific choice of language models.