Reviewerly/exHarmony huggingface dataset:
Structural Semantic Adversarial Active Learning for Image Captioning

Most image captioning models achieve superior performances with the help of large-scale surprised training data, but it is prohibitively costly to label the image captions. To solve this problem, we propose a structural semantic adversarial active learning (SSAAL) model that leverages both visual and textual information for deriving the most representative samples while maximizing the image captioning performance. SSAAL consists of a semantic constructor, a snapshot& caption (SC) supervisor, and a labeled/unlabeled state discriminator. The constructor is designed to generate a structural semantic representation describing the objects, attributes and object relationships in the image. The SC supervisor is proposed to supervise this representation at the word-level and sentence-level in a multi-task learning manner, which directly relates the representation to ground-truth captions and updates it in the caption generating process. Finally, we introduce a state discriminator to predict the sample state and select images with sufficient semantic and fine-grained diversity. Extensive experiments on standard captioning dataset show that our model outperforms other active learning methods and achieves a competitive performance even though selecting a small amount of samples.
Efficient Implementation of ARX-Based Block Ciphers on 8-Bit AVR Microcontrollers

As the development of Internet of Things (IoT), the data exchanged through the network has significantly increased. To secure the sensitive data with user’s personal information, it is necessary to encrypt the transmitted data. Since resource-constrained wireless devices are typically used for IoT services, it is required to optimize the performance of cryptographic algorithms which are computation-intensive tasks. In this paper, we present efficient implementations of ARX-based Korean Block Ciphers (HIGHT and LEA) with CounTeR (CTR) mode of operation, and CTR_DRBG, one of the most widely used DRBGs (Deterministic Random Bit Generators), on 8-bit AVR Microcontrollers (MCUs). Since 8-bit AVR MCUs are widely used for various types of IoT devices, we select it as the target platform in this paper. We present an efficient implementation of HIGHT and LEA by making full use of the property of CTR mode, where the nonce value is fixed, and only the counter value changes during the encryption. On our implementation, the cost of additional function calls occurred by the generation of look-up table can be reduced. With respect to CTR_DRBG, we identified several parts that do not need to be computed. Thus, precomputing those parts in offline and using them online can result in performance improvements for CTR_DRBG. Furthermore, we applied several optimization techniques by making full use of target devices’ characteristics with AVR assembly codes on 8-bit AVR MCUs. Our proposed table generation way can reduce the cost for building a precomputation table by around 6.7% and 9.1% in the case of LEA and HIGHT, respectively. Proposed implementations of LEA and HIGHT with CTR mode on 8-bit AVR MCUs provide 6.3% and 3.8% of improved performance, compared with the previous best results, respectively. Our implementations are the fastest compared to previous LEA and HIGHT implementations on 8-bit AVR MCUs. In addition, the proposed CTR_DRBG implementations on AVR provide better performance by 37.2% and 8.7% when the underlying block cipher is LEA and HIGHT, respectively.
Contesting control: journeys through surrender, self-awareness and looseness of control in embodied interaction

KEYWORDS: PerformanceMultimodal UINew UIBrain-Computer InteractionHuman-Robot Interactionarts and mediaHCI design theoryEmbodied InteractionAutonomous Systems
Learning Hierarchical Graph for Occluded Pedestrian Detection

Although pedestrian detection has made significant progress with the help of deep convolution neural networks, it is still a challenging problem to detect occluded pedestrians since the occluded ones can not provide sufficient information for classification and regression. In this paper, we propose a novel Hierarchical Graph Pedestrian Detector (HGPD), which integrates semantic and spatial relation information to construct two graphs named intra-proposal graph and inter-proposal graph, without relying on extra cues w.r.t visible regions. In order to capture the occlusion patterns and enhance features from visible regions, the intra-proposal graph considers body parts as nodes and assigns corresponding edge weights based on semantic relations between body parts. On the other hand, the inter-proposal graph adopts spatial relations between neighbouring proposals to provide additional proposal-wise context information for each proposal, which alleviates the lack of information caused by occlusion. We conduct extensive experiments on standard benchmarks of CityPersons and Caltech to demonstrate the effectiveness of our method. On CityPersons, our approach outperforms the baseline method by a large margin of 5.24pp on the heavy occlusion set, and surpasses all previous methods; on Caltech, we establish a new state of the art of 3.78% MR. Code is available at https://github.com/ligang-cs/PedestrianDetection-HGPD.
Realistic Video Summarization through VISIOCITY

Automatic video summarization is still an unsolved problem due to several challenges. We take steps towards making it more realistic by addressing the following challenges. Firstly, the currently available datasets either have very short videos or have few long videos of only a particular type. We introduce a new benchmarking dataset called VISIOCITY which comprises of longer videos across six different categories with dense concept annotations capable of supporting different flavors of video summarization and other vision problems. Secondly, for long videos, human reference summaries, necessary for supervised video summarization techniques, are difficult to obtain. We present a novel recipe based on pareto optimality to automatically generate multiple reference summaries from indirect ground truth present in VISIOCITY. We show that these summaries are at par with human summaries. Thirdly, we demonstrate that in the presence of multiple ground truth summaries (due to the highly subjective nature of the task), learning from a single combined ground truth summary using a single loss function is not a good idea. We propose a simple recipe VISIOCITY-SUM to enhance an existing model using a combination of losses and demonstrate that it beats the current state of the art techniques. We also present a study of different desired characteristics of a good summary and demonstrate that a single measure (say F1) to evaluate a summary, as is the current typical practice, falls short in some ways. We propose an evaluation framework for better quantitative assessment of summary quality which is closer to human judgment than a single measure. We report the performance of a few representative techniques of video summarization on VISIOCITY assessed using various measures and bring out the limitation of the techniques and/or the assessment mechanism in modeling human judgment and demonstrate the effectiveness of our evaluation framework in doing so.
Self-supervised representation learning by predicting visual permutations

We propose a self-supervised learning method to uncover the spatial or temporal structure of visual data by identifying the position of a patch within an image or the position of a video frame over time, which is related to Jigsaw puzzle reassembly problem in previous works. A Jigsaw puzzle can be seen as a shuffled sequence, which is generated by shuffling image patches or video frames according to an unknown permutation. The task of predicting the visual permutations can be used to train a learning system to capture structural information which is important for semantic-level tasks, such as object recognition and action recognition. To this end, we propose a multi-task learning framework where a group of principal tasks aims to predict the index of each sample in the original sequence, and a group of auxiliary tasks aims to predict the spatial or temporal relation of adjacent samples in the shuffled sequence. Our scheme can handle the whole space of permutations and is fairly scalable, and it is also generic to solve many problems such as self-supervised representation learning, relative attributes, and learning to rank. Our method achieves state-of-the-art performance on the STL-10 benchmarks for unsupervised representation learning, and it is competitive with state-of-the-art performance on UCF-101 and HMDB-51 as a pretraining method for action recognition. In addition, we apply the proposed method on age comparison task to prove it is generic to solve ranking problems.
Research and Implementation of Intelligent Decision Based on a Priori Knowledge and DQN Algorithms in Wargame Environment

The reinforcement learning problem of complex action control in a multi-player wargame has been a hot research topic in recent years. In this paper, a game system based on turn-based confrontation is designed and implemented with state-of-the-art deep reinforcement learning models. Specifically, we first design a Q-learning algorithm to achieve intelligent decision-making, which is based on the DQN (Deep Q Network) to model complex game behaviors. Then, an a priori knowledge-based algorithm PK-DQN (Prior Knowledge-Deep Q Network) is introduced to improve the DQN algorithm, which accelerates the convergence speed and stability of the algorithm. The experiments demonstrate the correctness of the PK-DQN algorithm, it is validated, and its performance surpasses the conventional DQN algorithm. Furthermore, the PK-DQN algorithm shows effectiveness in defeating the high level of rule-based opponents, which provides promising results for the exploration of the field of smart chess and intelligent game deduction.
OPHiForest: Order Preserving Hashing Based Isolation Forest for Robust and Scalable Anomaly Detection

Anomaly detection is one of the most important data mining tasks in many real-life applications such as network intrusion detection for cybersecurity and medical diagnosis for healthcare. In the big data era, these applications demand fast and versatile anomaly detection capability to handle various types of increasingly huge-volume data. However, existing detection methods are either slow due to high computational complexity, or unable to deal with complicated anomalies like local anomalies. In this paper, we propose a novel anomaly detection method named OPHiForest with the use of the order preserving hashing based isolation forest. The core idea is to learn the information from data to construct better isolation forest structure than the state-of-the-art methods like iForest and LSHiForest, which can achieve robust detection of various anomaly types. We design a fast two-step learning process for the order preserving hashing scheme. This leads to stronger order preservation for better hashing, and therefore enhances anomaly detection robustness and accuracy. Extensive experiments on both synthetic and real-world data sets demonstrate that our method is highly robust and scalable.
Do People and Neural Nets Pay Attention to the Same Words

We investigated how users evaluate passage-length answers for non-factoid questions. We conduct a study where answers were presented to users, sometimes shown with automatic word highlighting. Users were tasked with evaluating answer quality, correctness, completeness, and conciseness. Words in the answer were also annotated, both explicitly through user mark up and implicitly through user gaze data obtained from eye-tracking. Our results show that the correctness of an answer strongly depends on its completeness, conciseness is less important.
Noise Adaptive Wiener Filtering of Images

In this paper we propose a technique for application of the Wiener filter over images with unknown level of present Gaussian noise. The variance of the noise is predicted by exploring the intensity distribution of relatively homogenous areas from the image and then passed to the Wiener filter as an input argument prior to applying it in contrast to its traditional implementation. Also, extensive study has been made on the influence of the filter mask size over the quality of processed images given this approach. Promising results are gathered with comparison to the original implementation of the Wiener filter.
In-Memory Data Anonymization Using Scalable and High Performance RDD Design

Recent studies in data anonymization techniques have primarily focused on MapReduce. However, these existing MapReduce based approaches often suffer from many performance overheads due to their inappropriate use of data allocation, expensive disk I/O access and network transfer, and no support for iterative tasks. We propose “SparkDA” which is a new novel anonymization technique that is designed to take the full advantage of Spark platform to generate privacy-preserving anonymized dataset in the most efficient way possible. Our proposal offers a better partition control, in-memory operation and cache management for iterative operations that are heavily utilised for data anonymization processing. Our proposal is based on Spark’s Resilient Distributed Dataset (RDD) with two critical operations of RDD, such as FlatMapRDD and ReduceByKeyRDD, respectively. The experimental results demonstrate that our proposal outperforms the existing approaches in terms of performance and scalability while maintaining high data privacy and utility levels. This illustrates that our proposal is capable to be used in a wider big data applications that demands privacy.
poimo

Despite the recent growth in popularity of personal mobility devices (e.g., e-scooters and e-skateboards), they still suffer from limited safety and narrow design form factors, due to their rigid structures. On the other hand, inflatable interfaces studied in human-computer interaction can achieve large volume change by simple inflation/deflation. Inflatable structure also offers soft and safe interaction owing to material compliance and diverse fabrication methods that lead to a wide range of forms and aesthetics. In this paper, we propose poimo, a new family of POrtable and Inflatable MObility devices, which consists of inflatable frames, inflatable wheels, and inflatable steering mechanisms made of a mass-manufacturable material called drop-stitch fabric. First, we defined the basic material properties of a drop-stitch inflatable structure that is sufficiently strong to carry a person while simultaneously allowing soft deformation and deflation for storage and portability. We then implemented an interactive design system that can scan the user's desired riding posture to generate a customized personal mobility device and can add the user's shape and color preferences. To demonstrate the custom-design capability and mobility, we designed several 3D models using our system and built physical samples for two basic templates: a motorcycle and a wheelchair. Finally, we conducted an online user study to examine the usability of the design system and share lessons learned for further improvements in the design and fabrication of poimo.
An Automatic Detection and Analysis of the Bitcoin Generator Scam

We investigate what we call the "Bitcoin Generator Scam" (BGS), a simple system in which the scammers promise to "generate" new bitcoins using the ones that were sent to them. A typical offer will suggest that, for a small fee, one could receive within minutes twice the amount of bitcoins submitted. BGS is clearly not a very sophisticated attack. The modus operandi is simply to put up some web page on which to find the address to send the money and wait for the payback. The pages are then indexed by search engines, and ready to find for victims looking for free bitcoins. We describe here a generic system to find and analyze scams such as BGS. We have trained a classifier to detect these pages, and we have a crawler searching for instances using a series of search engines. We then monitor the instances that we find to trace payments and bitcoin addresses that are being used over time. Unlike most bitcoin-based scam monitoring systems, we do not rely on analyzing transactions on the blockchain to find scam instances. Instead, we proactively find these instances through the web pages advertising the scam. Thus our system is able to find addresses with very few transactions, or even none at all. Indeed, over half of the addresses that have eventually received funds were detected before receiving any transactions. The data for this paper was collected over four months, from November 2019 to February 2020. We have found more than 1,300 addresses directly associated with the scam, hosted on over 500 domains. Overall, these addresses have received (at least) over 5 million USD to the scam, with an average of 47.3 USD per transaction.
Performance Overhead of Atomic Crosschain Transactions

Atomic Crosschain Transaction technology allows composable programming across permissioned Ethereum blockchains. It allows for inter-contract and inter-blockchain function calls that are both synchronous and atomic: if one part fails, the whole call graph of function calls is rolled back. This paper analyses the processing overhead of using this technique compared to using multiple standard non-atomic single blockchain transactions. The additional processing is analysed for three scenarios involving multiple blockchains: the Hotel - Train problem, Supply Chain with Provenance, and an Oracle. The technology is shown to reduce the performance of Hyperledger Besu from 375 tps to 39.5 tps if all transactions are instigated on one node, or approaching 65.2 tps if the transactions are instigated on a variety of nodes, for the Hotel-Train scenario.
Distance Estimation of Preceding Vehicle Based on Mono Vision Camera and Artificial Neural Networks

Most of the research in automotive literature has been focused on the object detection task because of which distance estimation between vehicles has gained less attention in the computer vision. Distance between vehicles is foremost important to prevent accidents and enhance vehicle safety. To keep a safe distance between vehicles and avoid the traffic accidents, traditional method of inverse perspective mapping algorithm is used. It is observed that the algorithm resulted in poor performance for far distance objects. To solve the problem of high error rate of preceding vehicle distance measurement of driver assistant system, this paper proposes an Artificial Neural Network (ANN) based algorithm, which is used to calculate the distance of preceding vehicle from camera images. The proposed model uses a supervised learning algorithm, YOLO (you look only once) object detector for vehicle detection. The input and output attributes were evaluated from the object bounding boxes, which is obtained from the object detector. The proposed ANN based system was estimated on the straight road images. The experiments conducted on KITTI benchmark, indicate that the proposed method has reduced the error rate to 2% compared with other existing methods.
Set-Based Particle Swarm Optimization for Portfolio Optimization

Portfolio optimization is a complex real-world problem where assets are selected such that profit is maximized while risk is simultaneously minimized. In recent years, nature-inspired algorithms have become a popular choice for efficiently identifying optimal portfolios. This paper introduces such an algorithm that, unlike previous algorithms, uses a set-based approach to reduce the dimensionality of the problem and to determine the appropriate budget al.location for each asset. The results show that the proposed approach is capable of obtaining good quality solutions, while being relatively fast.
Leukocyte subtypes identification using bilinear self-attention convolutional neural network

Effective identification of leukocyte subtypes in microscopic images can help doctors diagnose diseases more accurately. Previous studies have achieved well performance by using segmentation techniques for multi-step processing. However, this increases the complexity of the whole identification process. In this paper, we proposed a novel model structure that can be trained end-to-end. The model combines attention mechanisms to emphasize the most discriminative features, and bilinear strategy to capture the interactions between features. We called this model Bilinear Self-Attention Network (BSA-Net). BSA-Net directly performs leukocyte subtypes identification in a one-step manner, which not only reduces complexity, but also achieves higher accuracy. To better understand the impact of the attention mechanism, we visualized the attention feature map in the BSA-Net model. Experiments demonstrated the effectiveness of our proposed method, which can meet the requirements of doctors for the accuracy and timeliness of cell identification results.
FPGA Accelerator for Stereo Vision using Semi-Global Matching through Dependency Relaxation

In this paper, we propose a fully parallel and pipelined architecture for stereo vision on FPGAs using Semi-Global Matching with Census Transform being used underneath. Further, we extend the above streaming architecture so that multiple pixels can be processed in a data parallel fashion. We expose this data parallelism through dependency relaxation. This establishes a trade-off between accuracy and throughput of the hardware. We tested the proposed architecture on Virtex-7 FPGA using KITTI 2012 and KITTI 2015 datasets. On images of resolution 1280x960, with 64 disparity levels, we are able to run our hardware design at 100 MHz. At this frequency, our design is able to process 322 frames per second which is 1.6 times faster than the state-of-the-art SGM implementation on FPGA. Our system can be scaled to a higher resolution image.
How long do Junior Developers take to Remove Technical Debt Items?

Background. Software engineering is one of the engineering fields with the highest inflow of junior engineers. Tools that utilize source code analysis to provide feedback on internal software quality, i.e. Technical Debt (TD), are valuable to junior developers who can learn and improve their coding skills with minimal consultations with senior colleagues. Objective. We aim at understating which SonarQube TD items junior developers prioritize during the refactoring and how long they take to refactor them. Method. We designed a case study with replicated design and we conducted it with 185 junior developers in two countries, that developed 23 projects with different programming languages and architectures. Results. Junior developers focus homogeneously on different types of TD items. Moreover, they can refactor items in a fraction of the estimated time, never spending more than 50% of the time estimated by SonarQube. Conclusion. Junior Developers appreciate the usage of SonarQube and considered as a useful tool. Companies might ask junior developers to quickly clean their code.
Automatic Generation of Super Mario Levels via Graph Grammars

Automatically generating game levels using Procedural Content Generation (PCG) is a challenging problem because of the necessity of attending both functional and nonfunctional requirements. In the particular example of level generation for Super Mario, Machine Learning approaches, such as GANs and Reinforcement Learning, have shown promise. However, these black-box approaches usually are not explainable, and thus difficult to integrate with human designers. We propose a different level-generation system that uses a reachable graph structure, automatic detection of level structures, and formal graph grammars. A human designer can also easily add or remove structures to use the system as a level co-creation tool. An experimental analysis shows that the proposed system can generate playable and visually pleasing levels, while revealing some limitations of current approaches on platformer level generation, such as the generation of backtracking segments.