Reviewerly/exHarmony huggingface dataset:
VisualHow: Multimodal Problem Solving

Recent progress in the interdisciplinary studies of computer vision (CV) and natural language processing (NLP) has enabled the development of intelligent systems that can describe what they see and answer questions accordingly. However, despite showing usefulness in performing these vision-language tasks, existing methods still struggle in understanding real-life problems (i.e., how to do something) and suggesting step-by-step guidance to solve them. With an overarching goal of developing intelligent systems to assist humans in various daily activities, we propose VisualHow, a free-form and open-ended research that focuses on understanding a real-life problem and deriving its solution by incorporating key components across multiple modalities. We develop a new dataset with 20,028 real-life problems and 102,933 steps that constitute their solutions, where each step consists of both a visual illustration and a textual description that guide the problem solving. To establish better understanding of problems and solutions, we also provide annotations of multimodal attention that localizes important components across modalities and solution graphs that encapsulate different steps in structured representations. These data and annotations enable a family of new vision-language tasks that solve real-life problems. Through extensive experiments with representative models, we demonstrate their effectiveness on training and testing models for the new tasks, and there is significant scope for improvement by learning effective attention mechanisms. Our dataset and models are available at https://github.com/formidify/VisualHow.
Lower Bound on SNARGs in the Random Oracle Model

Succinct non-interactive arguments (SNARGs) have become a fundamental primitive in the cryptographic community. The focus of this work is constructions of SNARGs in the Random Oracle Model (ROM). Such SNARGs enjoy post-quantum security and can be deployed using lightweight cryptography to heuristically instantiate the random oracle. A ROM-SNARG is $$(t,\varepsilon )$$ -sound if no $$t$$ -query malicious prover can convince the verifier to accept a false statement with probability larger than $$\varepsilon $$ . Recently, Chiesa-Yogev (CRYPTO ’21) presented a ROM-SNARG of length $${\varTheta }(\log (t/\varepsilon ) \cdot \log t)$$ (ignoring $$\log n$$ factors, for n being the instance size). This improvement, however, is still far from the (folklore) lower bound of $$\varOmega (\log (t/\varepsilon ))$$ . Assuming the randomized exponential-time hypothesis, we prove a tight lower bound of $${\varOmega }(\log (t/\varepsilon ) \cdot \log t)$$ for the length of $$(t,\varepsilon )$$ -sound ROM-SNARGs. Our lower bound holds for constructions with non-adaptive verifiers and strong soundness notion called salted soundness, restrictions that hold for all known constructions (ignoring contrived counterexamples). We prove our lower bound by transforming any short ROM-SNARG (of the considered family) into a same length ROM-SNARG in which the verifier asks only a few oracles queries, and then apply the recent lower bound of Chiesa-Yogev (TCC ’20) for such SNARGs.
An Ensemble Learning Algorithm Based on Density Peaks Clustering and Fitness for Imbalanced Data

In view of the low classification accuracy of the minority class in imbalanced data, an algorithm called DPF-EL (Density Peaks and Fitness Ensemble Learning) based on density peaks clustering and fitness is proposed. Firstly, this method uses the density peaks clustering algorithm to divide the majority class into different sub-clusters, the local density calculated in the clustering process is used to assign weights to each sub-cluster, and the number of under-samplings is determined by the weights. Secondly, the concept of fitness is introduced into the sub-clusters. The selection probability of the samples is calculated according to the size of their fitness, and the majority class is under-sampled based on the selection probability. Finally, combined with boosting algorithm, iterative training is performed on the balanced data set. Experimental tests were conducted with KEEL imbalanced data sets, and the experimental results show that the performance of DPF-EL algorithm is better than other algorithms, which indicates the feasibility of the proposed algorithm.
Implicit Event Argument Extraction With Argument-Argument Relational Knowledge

As a challenging sub-task of event argument extraction, implicit event argument extraction seeks to identify document-level arguments that play direct or implicit roles in a given event. Prior work mainly focuses on capturing direct relations between arguments and the event trigger; however, the lack of reasoning ability imposes limitations to the extraction of implicit arguments. In this work, we propose an <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> rgument-argument <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</b> elation-enhanced <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">E</b> vent <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> rgument extraction (AREA) learning framework to tackle this issue through reasoning in event frame-level scope. The proposed method leverages related arguments of the expected one as clues, and utilizes such argument-argument dependencies to guide the reasoning process. To bridge the distribution gap between oracle knowledge used in the training phase and the imperfect related arguments in the test stage, we introduce a conventional knowledge distillation strategy to drive a final model that can work without extra inputs by mimicking the behaviour of a well-informed teacher model. In addition, considering that conventional knowledge distillation methods transfer knowledge individually, we integrate it with a novel relational knowledge distillation mechanism to explicitly capture the structural mutual argument-argument relation. Moreover, since the training process is not compatible with the real situation, a curriculum learning method is further introduced to make the training process smoother. Experimental results demonstrate that the learning framework obtains state-of-the-art performance on the RAMS and Wikievents datasets. Ablation study and further discussion also show it could handle long-range dependency and implicit argument problems effectively.
Efficient Learning of Inverse Dynamics Models for Adaptive Computed Torque Control

Modelling robot dynamics accurately is essential for control, motion optimisation and safe human-robot collaboration. Given the complexity of modern robotic systems, dynamics modelling remains non-trivial, mostly in the presence of compliant actuators, mechanical inaccuracies, friction and sensor noise. Recent efforts have focused on utilising datadriven methods such as Gaussian processes and neural networks to overcome these challenges, as they are capable of capturing these dynamics without requiring extensive knowledge beforehand. While Gaussian processes have shown to be an effective method for learning robotic dynamics with the ability to also represent the uncertainty in the learned model through its variance, they come at a cost of cubic time complexity rather than linear, as is the case for deep neural networks. In this work, we leverage the use of deep kernel models, which combine the computational efficiency of deep learning with the nonparametric flexibility of kernel methods (Gaussian processes), with the overarching goal of realising an accurate probabilistic framework for uncertainty quantification. Through using the predicted variance, we adapt the feedback gains as more accurate models are learned, leading to low-gain control without compromising tracking accuracy. Using simulated and real data recorded from a seven degree-of-freedom robotic manipulator, we illustrate how using stochastic variational inference with deep kernel models increases compliance in the computed torque controller, and retains tracking accuracy. We empirically show how our model outperforms current state-of-the-art methods with prediction uncertainty for online inverse dynamics model learning, and solidify its adaptation and generalisation capabilities across different setups.
Personalized Learning Using Kernel Methods and Random Fourier Features

Distributed learning algorithms have become an increasingly popular research topic due to the increasing proliferation of IoT devices and sensor networks with edge processing capabilities. This paper focuses on a distributed learning algorithm, Personalized Learning. In Personalized Learning, a population of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$N$</tex> edge devices learns its similarities while preserving its learnings individually. Personalized Learning consists of a two-step iterative approach: <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$N$</tex> edge processors are learning for each <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$L$</tex> “canonical” model, and the characteristics of canonical models are estimated based on the learned edges. In earlier works, linear and logistic regression models were used to estimate characteristics in the first step. This work extends the previous work on the framework for personalized learning using non-linear data. This research improves the computational speed and performance of the current framework using Kernel Methods by Random Fourier Features for learning the first layer and compares these learning methods with linear models in simulation studies.
Why Don’t XAI Techniques Agree? Characterizing the Disagreements Between Post-hoc Explanations of Defect Predictions

Machine Learning (ML) based defect prediction models can be used to improve the reliability and overall quality of software systems. However, such defect predictors might not be deployed in real applications due to the lack of transparency. Thus, recently, application of several post-hoc explanation methods (e.g., LIME and SHAP) have gained popularity. These explanation methods can offer insight by ranking features based on their importance in black box decisions. The explainability of ML techniques is reasonably novel in the Software Engineering community. However, it is still unclear whether such explainability methods genuinely help practitioners make better decisions regarding software maintenance. Recent user studies show that data scientists usually utilize multiple post-hoc explainers to understand a single model decision because of the lack of ground truth. Such a scenario causes disagreement between explainability methods and impedes drawing a conclusion. Therefore, our study first investigates three disagreement metrics between LIME and SHAP explanations of 10 defect-predictors, and exposes that disagreements regarding the rankings of feature importance are most frequent. Our findings lead us to propose a method of aggregating LIME and SHAP explanations that puts less emphasis on these disagreements while highlighting the aspect on which explanations agree.
Source-Free Unsupervised Domain Adaptation in Imbalanced Datasets

Existing source-free unsupervised domain adaptation uses the source model and unlabeled target data to train the target model under the assumption that the dataset is relatively balanced. However, in many practical applications, the class distribution is often skewed. Existing works tackle this problem mainly by oversampling the minority classes or undersampling the majority classes, which are not applicable when data are unlabeled. To address these challenges, we estimate class distribution by summing over model prediction probabilities to more flexibly tackle the uncertainty in an imbalanced dataset and employ information entropy of the classification results to select reliable balancing samples. Furthermore, since information entropy of the classification results tends to decrease as the model iterates in training, we propose Dynamic Entropy-based Balance (DEB) to more precisely determine the threshold for reliable balancing sample selection. Experimental results on several real datasets validate the effectiveness of our method.
Tiresias

To efficiently store and query a DBMS, administrators must select storage and indexing configurations. For example, one must decide whether data should be stored in rows or columns, in-memory or on disk, and which columns to index. These choices can be challenging to make for workloads that are mixed requiring hybrid transactional and analytical processing (HTAP) support. There is growing interest in system designs that can adapt how data is stored and indexed to execute these workloads efficiently. We present Tiresias , a predictor that learns the cost of data accesses and predicts their latency and likelihood under different storage scenarios. Tiresias makes these predictions by collecting observed latencies and access histories to build predictive models in an online manner, enabling autonomous storage and index adaptation. Experimental evaluation shows the benefits of predictive adaptation and the trade-offs for different predictive techniques.
Blockchain and Logistics

The participants in the logistics process are different entities from different areas of the logistics. When they work together to establish productive relationships, a massive cost is needed to solve the trust issues, such as operational costs of service quality, billing reconciliation costs, and audit management costs for logistics documents, etc. Blockchain technology can effectively solve the trust problem in large logistics, and can facilitate the realization of large-scale, low-cost and high-trust logistics platforms. This chapter introduces the trust problems in the logistics industry, as well as the solutions based on the blockchain. It also presents the application scenarios and practices to help better understand the application of the blockchain technology in the logistics industry.
Quid pro quo

We explore the role of reciprocity in code review processes. Reciprocity manifests itself in two ways: 1) reviewing code for others translates to accepted code contributions, and 2) having contributions accepted increases the reviews made for others. We use vector autoregressive (VAR) models to explore the causal relation between reviews performed and accepted contributions. After fitting VAR models for 24 active open-source developers, we found evidence of reciprocity in 6 of them. These results suggest reciprocity does play a role in code review, that can potentially be exploited to increase reviewer participation.
GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-training

Most of the currently existing vision and language pre-training (VLP) methods have mainly focused on how to extract and align vision and text features. In contrast to the mainstream VLP methods, we highlight that two routinely applied steps during pre-training have crucial impact on the performance of the pre-trained model: in-batch hard negative sampling for image-text matching (ITM) and assigning the large masking probability for the masked language modeling (MLM). After empirically showing the unexpected effectiveness of above two steps, we systematically devise our GRIT-VLP, which adaptively samples mini-batches for more effective mining of hard negative samples for ITM while maintaining the computational cost for pre-training. Our method consists of three components: 1) GRouped mIni-baTch sampling (GRIT) strategy that collects similar examples in a mini-batch, 2) ITC consistency loss for improving the mining ability, and 3) enlarged masking probability for MLM. Consequently, we show our GRIT-VLP achieves a new state-of-the-art performance on various downstream tasks with much less computational cost. Furthermore, we demonstrate that our model is essentially in par with ALBEF, the previous state-of-the-art, only with one-third of training epochs on the same training data. Code is available at https://github.com/jaeseokbyun/GRIT-VLP .
Temporal Shift and Attention Modules for Graphical Skeleton Action Recognition

Skeletons, consisting of joint positions and connections between them, are an important representation for modeling human bodies in image frames. Compared with understanding RGB videos, recognizing actions from the skeletons removes the biases of background and body shapes. Researchers use spatial-temporal graphs to model the skeleton sequences. These methods weigh all frames in the sequence equally even though many of the frames may not be useful for action and prediction and dilute the influence of important frames. Also, the temporal graph focuses on understanding only the low-level feature of the joints for the motion of the skeleton. In this paper, we introduce two modules, temporal shift module and temporal attention module that can be added to graph convolution networks for skeleton action recognition. Temporal attention module focuses on keyframes for making predictions, and temporal shift module helps to exchange the high-level features between different frames along the temporal dimension besides the local patterns. We evaluate the two modules with two existing skeleton action recognition networks, ST-GCN and MS-G3D, on three public datasets and show better results than the original methods.
UAV Formation Obstacle Avoidance Based on Improved Consistency Algorithm

Aiming at the formation obstacle avoidance problem of multiple Unmanned Aerial Vehicles (UAV) system in complex environment, this paper designs a virtual agent based on consistent formation algorithm, so as to minimize the impact of obstacle avoidance process on formation. When an agent needs to avoid obstacles, it will generate a virtual agent to maintain consistent communication with other UAVs, which is conducive to maintaining the consistency of the whole formation. Compared with the classical artificial potential field obstacle avoidance method, the simulation results show that the proposed method in this paper can effectively realize formation obstacle avoidance, and has more advantages in formation maintenance after obstacle avoidance.
Density Aware Blue-Noise Sampling on Graphs

Efficient sampling of graph signals is essential to graph signal processing. Recently, blue-noise was introduced as a sampling method that maximizes the separation between sampling nodes leading to high-frequency dominance patterns, and thus, to high-quality patterns. Despite the simple inter-pretation of the method, blue-noise sampling is restricted to approximately regular graphs. This study presents an extension of blue-noise sampling that allows the application of the method to irregular graphs. Before sampling with a blue-noise algorithm, the approach regularizes the weights of the edges such that the graph represents a regular structure. Then, the resulting pattern adapts the node's distribution to the local density of the nodes. This work also uses an approach that minimizes the strength of the high-frequency components to recover approximately bandlimited signals. The experimental results show that the proposed methods have superior performance compared to the state-of-the-art techniques.
Simplifying Knowledge-Aware Aggregation for Knowledge Graph Collaborative Filtering

Incorporating knowledge graph (KG) for recommendation has been well considered in recent researches since it can alleviate the sparsity and cold-start problem of collaborative filtering. To capture the rich semantics of knowledge graph, existing KG-based models utilize graph neural networks (GNNs). However, we empirically find that the feature transformation and nonlinear activation designs in GNN contribute little to the recommendation performance. We propose simplified knowledge-aware attention network (SKAN) that simplifies the knowledge-aware aggregation by removing the two designs. To ensure the personalization during propagation, we apply weighted aggregation with user-specific attentions. We further aggregate the interacted items of users to enhance the user representation learning. We apply the proposed model on three real-world datasets, and the empirical results suggest that simplified knowledge-aware attention network (SKAN) significantly outperforms several compelling state-of-the-art baselines.
AME: Attention and Memory Enhancement in Hyper-Parameter Optimization

Training Deep Neural Networks (DNNs) is inherently subject to sensitive hyper-parameters and untimely feedbacks of performance evaluation. To solve these two difficulties, an efficient parallel hyper-parameter optimization model is proposed under the framework of Deep Reinforcement Learning (DRL). Technically, we develop Attention and Memory Enhancement (AME), that includes multi-head attention and memory mechanism to enhance the ability to capture both the short-term and long-term relationships between different hyper-parameter configurations, yielding an attentive sampling mechanism for searching high-performance configurations embedded into a huge search space. During the optimization of transformer-structured configuration searcher, a conceptually intuitive yet powerful strategy is applied to solve the problem of insufficient number of samples due to the untimely feedback. Experiments on three visual tasks, including image classification, object detection, semantic segmentation, demonstrate the effectiveness of AME.
Bulls-Eye: Active Few-Shot Learning Guided Logic Synthesis

Generating suboptimal synthesis transformation sequences (“synthesis recipe”) is an important problem in logic synthesis. Manually crafted synthesis recipes have poor quality. State-of-the art machine learning (ML) works to generate synthesis recipes do not scale to large netlists as the models need to be trained from scratch, for which training data is collected using time-consuming synthesis runs. We propose a new approach, Bulls-Eye, that fine-tunes a pretrained model on past synthesis data to accurately predict the quality of a synthesis recipe for an unseen netlist. Our approach achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula> – <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$30\times $ </tex-math></inline-formula> runtime improvement and generates synthesis recipes achieving close to 95% quality-of-result (QoR) compared to conventional techniques using actual synthesis runs. We show our QoR beat state-of-the-art approaches on various benchmarks.
The STEM teachers' course "Development of virtual and augmented reality software"

The experience of creating and utilizing learning materials for the construction of virtual reality (VR) and augmented reality (AR) systems for aspiring STEM teachers is examined in the paper. For the creation of instructional materials, the Unity visual design environment, Visual Studio for programming, and the Google VR and Vuforia platforms were selected. The page provides a description of the course's material. The study examines the course participants' perceptions. As a qualitative data collecting technique, interviews were used to gather the research data. To sum up, the course's creation encourages the growth of digital proficiency in the creation and application of cutting-edge teaching tools.