Reviewerly/exHarmony huggingface dataset:
Integrated multi-operand electro-optic logic gates for optical computing

Integrated optical computing has the potential to enhance the computation abilities in terms of computing speed and power consumption during the post-Moore's law era. As one of the widely investigated approaches, electro-optic logic uses electro-optic switches as the building blocks to achieve complex functions. In this paper, we propose and experimentally demonstrate a multioperand electro-optic logic gate that contains multiple active regions in a single gate to further increase the versatility of gates. Thus, one gate can be controlled by several electrical inputs simultaneously. The ability of multioperand logic gates is thoroughly explored. The detailed comparison shows that the proposed gate can largely improve the circuit performance in terms of the area, latency, power, and insertion loss. We believe the multioperand logic gate has the potential to contribute to a more compact and power-efficient integrated optical computing system.
Watch It Twice

With the rapid growth of video data and the increasing demands of various crossmodal applications such as intelligent video search and assistance towards visually-impaired people, video captioning task has received a lot of attention recently in computer vision and natural language processing fields. The state-of-the-art video captioning methods focus more on encoding the temporal information, while lacking effective ways to remove irrelevant temporal information and also neglecting the spatial details. In particular, the current unidirectional video encoder can be negatively affected by irrelevant temporal information, especially the irrelevant information at the beginning and at the end of a video. In addition, disregarding detailed spatial features may lead to incorrect word choices in decoding. In this paper, we propose a novel recurrent video encoding method and a novel visual spatial feature for the video captioning task. The recurrent encoding module encodes the video twice with a predicted key frame to avoid irrelevant temporal information often occurring at the beginning and at the end of a video. The novel spatial features represent spatial information from different regions of a video and provide the decoder with more detailed information. Experiments on two benchmark datasets show superior performance of the proposed method.
AR Graphic Representation of Musical Notes for Self-Learning on Guitar

Despite being one of the most commonly self-taught instruments, and despite the ready availability of significant amounts of didactic material, the guitar is a challenging instrument to learn. This paper proposes an application based on augmented reality (AR) that is designed to teach beginner students basic musical chords on the guitar, and provides details of the experimental study performed to determine whether the AR methodology produced faster results than traditional one-on-one training with a music teacher. Participants were divided into two groups of the same size. Group 1 consisted of 32 participants who used the AR app to teach themselves guitar, while Group 2, with a further 32 participants, received formal instruction from a music teacher. Results found no differences in learning times between the two groups based on the variables of method and gender. However, participant feedback suggested that there are advantages to the self-taught approach using AR that are worth considering. A system usability scale (SUS) questionnaire was used to measure the usability of the application, obtaining a score of 82.5, which was higher than the average of 68 that indicates an application to be good from a user experience point of view, and satisfied the purpose for which the application was created.
Cross-Car, Multiplayer Games for Semi-Autonomous Driving

We investigate and characterize a design space for in-car games based on a survey of previous work, and identify an opportunity for "cross-car" multiplayer games played among occupants in nearby cars. This is supported by innovations in automotive technology like autonomous driving, full-window heads-up displays, and ad hoc communication between vehicles. In a custom virtual reality driving simulator, we created three games to illustrate design dimensions: Killerball, a competitive free-for-all game; Billiards, a player versus player, massively multiplayer online game with player assists; and Decoration, an idle-style game with multiplayer resource management. A 12-participant evaluation with a semi-structured interview revealed a positive response to input controls and HUDs, and suggests game genres have a similar effect on time for an emergency driving takeover task. We used insights from our process and evaluation to formulate design considerations for future cross-car games.
Relations in Ontology-Driven Conceptual Modeling

For over a decade now, a community of researchers has contributed to the ontological foundations of Conceptual Modeling by participating to the development of the Unified Foundational Ontology (UFO) and the UFO-based modeling language OntoUML, which have been successfully employed in a number of different sectors. The empirical feedback from these experiences led us to reconsider UFO's theory of relations, proposing a new theory that has already been applied to model subtle notions in the business domain, such as value, risk, service, and contract. In this paper, we advance a first formal characterization of this new theory, which is then used to design a new metamodel for OntoUML.
You Only Recognize Once

Video text spotting is still an important research topic due to its various real-applications. Previous approaches usually fall into the four-staged pipeline: text detection in individual images, frame-wisely recognizing localized text regions, tracking text streams and generating final results with complicated post-processing skills, which might suffer from the huge computational cost as well as the interferences of low-quality text. In this paper, we propose a fast and robust video text spotting framework by only recognizing the localized text one-time instead of frame-wisely recognition. Specifically, we first obtain text regions in videos with a well-designed spatial-temporal detector. Then we concentrate on developing a novel text recommender for selecting the highest-quality text from text streams and only recognizing the selected ones. Here, the recommender assembles text tracking, quality scoring and recognition into an end-to-end trainable module, which not only avoids the interferences from low-quality text but also dramatically speeds up the video text spotting process. In addition, we collect a larger scale video text dataset (LSVTD) for promoting the video text spotting community, which contains 100 text videos from 22 different real-life scenarios. Extensive experiments on two public benchmarks show that our method greatly speeds up the recognition process averagely by 71 times compared with the frame-wise manner, and also achieves the remarkable state-of-the-art.
Question-Aware Tube-Switch Network for Video Question Answering

Video Question & Answering (VideoQA), a task to answer questions in videos, involves rich spatio-temporal content (e.g., appearance and motion) and requires multi-hop reasoning process. However, existing methods usually deal with appearance and motion separately and fail to synchronize the attentions on appearance and motion features, neglecting two key properties of video QA: (1) appearance and motion features are usually concomitant and complementary to each other at time slice level. Some questions rely on joint representations of both kinds of features at some point in the video; (2) appearance and motion have different importance in multi-step reasoning. In this paper, we propose a novel Question- Aware Tube-Switch Network (TSN) for video question answering which contains (1) a Mix module to synchronously combine the appearance and motion representation at time slice level, achieving fine-grained temporal alignment and correspondence between appearance and motion at every time slice and (2) a Switch mod- ule to adaptively choose appearance or motion tube as primary at each reasoning step, guiding the multi-hop reasoning process. To end-to-end train TSN, we utilize the Gumbel-Softmax strategy to account for the discrete tube-switch process. Extensive experimental results on two benchmarks: MSVD-QA and MSRVTT-QA, have demonstrated that the proposed TSN consistently outperforms state-of-the-art on all metrics.
Multi-modal Knowledge-aware Hierarchical Attention Network for Explainable Medical Question Answering

Online healthcare services can offer public ubiquitous access to the medical knowledge, especially with the emergence of medical question answering websites, where patients can get in touch with doctors without going to hospital. Explainability and accuracy are two main concerns for medical question answering. However, existing methods mainly focus on accuracy and cannot provide a good explanation for retrieved medical answers. This paper proposes a novelMulti-Modal Knowledge-aware Hierarchical Attention Network (MKHAN) to effectively exploit multi-modal knowledge graph (MKG) for explainable medical question answering. MKHAN can generate path representation by composing the structural, linguistics, and visual information of entities, and infer the underlying rationale of question-answer interactions by leveraging the sequential dependencies within a path from MKG. Furthermore, a novel hierarchical attention network is proposed to discriminate the salience of paths endowing our model with explainability. We build a large-scale multi-modal medical knowledge graph andtwo real-world medical question answering datasets, the experimental results demonstrate the superior performance on our approachcompared with the state-of-the-art methods.
EBPC: Extended Bit-Plane Compression for Deep Neural Network Inference and Training Accelerators

In the wake of the success of convolutional neural networks in image classification, object recognition, speech recognition, etc., the demand for deploying these compute-intensive ML models on embedded and mobile systems with tight power and energy constraints at low cost, as well as for boosting throughput in data centers, is growing rapidly.This has sparked a surge of research into specialized hardware accelerators.Their performance is typically limited by I/O bandwidth, power consumption is dominated by I/O transfers to off-chip memory, and on-chip memories occupy a large part of the silicon area.We introduce and evaluate a novel, hardware-friendly, and lossless compression scheme for the feature maps present within convolutional neural networks.We present hardware architectures and synthesis results for the compressor and decompressor in 65 nm.With a throughput of one 8-bit word/cycle at 600 MHz, they fit into 2.8 kGE and 3.0 kGE of silicon area, respectivelytogether the size of less than seven 8-bit multiply-add units at the same throughput.We show that an average compression ratio of 5.1× for AlexNet, 4× for VGG-16, 2.4× for ResNet-34 and 2.2× for MobileNetV2 can be achieved-a gain of 45-70% over existing methods.Our approach also works effectively for various number formats, has a low frame-to-frame variance on the compression ratio, and achieves compression factors for gradient map compression during training that are even better than for inference.
Reward-driven U-Net training for obstacle avoidance drone

Along with the fast progress in deep learning, an autonomous drone with obstacle avoidance capability has been studied mainly by two machine learning paradigms, i.e. supervised learning, and reinforcement learning. The former has some advantages since the trained network is light and fast, but it needs a large amount of data that requires laborious manual labeling. With the latter, such a drawback can be overcome as an agent learns by itself in a simulated environment, although the gap between the real and simulated one has to be minimized in the end. This study proposes a new framework where a supervised segmentation network is trained with labels made by an actor-critic network in a reward-driven manner, wherein this U-Net based network infers the next moving direction from the sequence of input images. For the actor-critic part, several recent policy gradient algorithms have been tested for controlling the drone with the continuous action space. After training in the Airsim simulation environment, the model is transferred to a Bebop drone flying in the real environment, built as a reconfigurable maze using panels and a hoop. The result suggests that our network enables the drone to navigate through the obstacles using only monocular RGB input in the trained environment as well as in the reconfigured ones without retraining.
Knowledge Distillation for Semi-supervised Domain Adaptation

In the absence of sufficient data variation (e.g., scanner and protocol variability) in annotated data, deep neural networks (DNNs) tend to overfit during training. As a result, their performance is significantly lower on data from unseen sources compared to the performance on data from the same source as the training data. Semi-supervised domain adaptation methods can alleviate this problem by tuning networks to new target domains without the need for annotated data from these domains. Adversarial domain adaptation (ADA) methods are a popular choice that aim to train networks in such a way that the features generated are domain agnostic. However, these methods require careful dataset-specific selection of hyperparameters such as the complexity of the discriminator in order to achieve a reasonable performance. We propose to use knowledge distillation (KD) – an efficient way of transferring knowledge between different DNNs – for semi-supervised domain adaption of DNNs. It does not require dataset-specific hyperparameter tuning, making it generally applicable. The proposed method is compared to ADA for segmentation of white matter hyperintensities (WMH) in magnetic resonance imaging (MRI) scans generated by scanners that are not a part of the training set. Compared with both the baseline DNN (trained on source domain only and without any adaption to target domain) and with using ADA for semi-supervised domain adaptation, the proposed method achieves significantly higher WMH dice scores.
3D Neighborhood Convolution: Learning Depth-Aware Features for RGB-D and RGB Semantic Segmentation

A key challenge for RGB-D segmentation is how to effectively incorporate 3D geometric information from the depth channel into 2D appearance features. We propose to model the effective receptive field of 2D convolution based on the scale and locality from the 3D neighborhood. Standard convolutions are local in the image space (u, v), often with a fixed receptive field of 3x3 pixels. We propose to define convolutions local with respect to the corresponding point in the 3D real world space (x, y, z), where the depth channel is used to adapt the receptive field of the convolution, which yields the resulting filters invariant to scale and focusing on the certain range of depth. We introduce 3D Neighborhood Convolution (3DN-Conv), a convolutional operator around 3D neighborhoods. Further, we can use estimated depth to use our RGB-D based semantic segmentation model from RGB input. Experimental results validate that our proposed 3DN-Conv operator improves semantic segmentation, using either ground-truth depth (RGB-D) or estimated depth (RGB).
SQR: In-network Packet Loss Recovery from Link Failures for Highly Reliable Datacenter Networks

In datacenter networks, flows need to complete as quickly as possible because the flow completion time (FCT) directly impacts user experience, and thus revenue. Link failures can have a significant impact on short latency-sensitive flows because they increase their FCTs by several fold. Existing link failure management techniques cannot keep the FCTs low under link failures because they cannot completely eliminate packet loss during such failures. We observe that to completely mask the effect of packet loss and the resulting long recovery latency, the network has to be responsible for packet loss recovery instead of relying on end-to-end recovery. To this end, we propose Shared Queue Ring (SQR), an on-switch mechanism that completely eliminates packet loss during link failures by diverting the affected flows seamlessly to alternative paths. We implemented SQR on a Barefoot Tofino switch using the P4 programming language. Our evaluation on a hardware testbed shows that SQR can completely mask link failures and reduce tail FCT by up to 4 orders of magnitude for latency-sensitive workloads.
Learning to Ask

Product search is generally recognized as the first and foremost stage of online shopping and thus significant for users and retailers of e-commerce. Most of the traditional retrieval methods use some similarity functions to match the user's query and the document that describes a product, either directly or in a latent vector space. However, user queries are often too general to capture the minute details of the specific product that a user is looking for. In this paper, we propose a novel interactive method to effectively locate the best matching product. The method is based on the assumption that there is a set of candidate questions for each product to be asked. In this work, we instantiate this candidate set by making the hypothesis that products can be discriminated by the entities that appear in the documents associated with them. We propose a Question-based Sequential Bayesian Product Search method, QSBPS, which directly queries users on the expected presence of entities in the relevant product documents. The method learns the product relevance as well as the reward of the potential questions to be asked to the user by being trained on the search history and purchase behavior of a specific user together with that of other users. The experimental results show that the proposed method can greatly improve the performance of product search compared to the state-of-the-art baselines.
Video Integrity Verification and GOP Size Estimation Via Generalized Variation of Prediction Footprint

The Variation of Prediction Footprint (VPF), formerly used in video forensics for double compression detection and GOP size estimation, is comprehensively investigated to improve its acquisition capabilities and extend its use to video sequences that contain bi-directional frames (B-frames). By relying on a universal rate-distortion analysis applied to a generic double compression scheme, we first explain the rationale behind the presence of the VPF in double compressed videos and then justify the need of exploiting a new source of information such as the motion vectors, to enhance the VPF acquisition process. Finally, we describe the shifted VPF induced by the presence of B-frames and detail how to compensate the shift to avoid misguided GOP size estimations. The experimental results show that the proposed Generalized VPF (G-VPF) technique outperforms the state of the art, not only in terms of double compression detection and GOP size estimation, but also in reducing computational time.
Plant Biology-Inspired Genetic Algorithm: Superior Efficiency to Firefly Optimizer

This chapter analytically compares the efficiency of the recent plant biology-inspired genetic algorithm (PBGA) and the firefly algorithm (FA) optimizer. The comparison is over a range of well-known critical benchmark test functions. Through statistical comparisons over the benchmark functions, the efficiency of PBGA has been evaluated versus FA as a well-known accurate meta-heuristic optimizer. Through a considerable number of Monte Carlo runs of searching for a solution by both optimizers, their performance has been statistically measured by several valid indices. In addition, the convergence curves give a visual comparison of both techniques where the stability, speed, and accuracy dominance of PBGA is clearly observable. However, in the case of benchmark function with smooth nature-like Rosenbrock, Sphere, and Dixon and Price, FA has better performance on average, while PBGA performance is still comparable to FA.
Dynamics and Optimization Control of a Robust Chaotic Map

Robust chaos in the discrete system is suggested to have practical as well as theoretical importance since it can obtain reliable operation in the chaotic mode.However, it receives only moderate attention and only focuses on a finite chaotic parameter space and small Lyapunov exponents.This paper introduces a two-dimensional smooth map and studies its robustness of chaos in the infinite parameter space.Then, a compound operation-based optimization control method is introduced to increase the map complexity in the measure of Lyapunov exponent.The introduced method is simple and provides a new pathway for exploring the robustness and complexity of discrete chaotic system.Finally, we design a chaosbased pseudo-random number generator (CPNG) based on the optimized robust chaotic map, and the careful analysis shows that the proposed CPNG has high quality of randomness and has passed the rigorous National Institute of Standards and Technology (NIST) test.
Phrase Localization Without Paired Training Examples

Localizing phrases in images is an important part of image understanding and can be useful in many applications that require mappings between textual and visual information. Existing work attempts to learn these mappings from examples of phrase-image region correspondences (strong supervision) or from phrase-image pairs (weak supervision). We postulate that such paired annotations are unnecessary, and propose the first method for the phrase localization problem where neither training procedure nor paired, task-specific data is required. Our method is simple but effective: we use off-the-shelf approaches to detect objects, scenes and colours in images, and explore different approaches to measure semantic similarity between the categories of detected visual elements and words in phrases. Experiments on two well-known phrase localization datasets show that this approach surpasses all weakly supervised methods by a large margin and performs very competitively to strongly supervised methods, and can thus be considered a strong baseline to the task. The non-paired nature of our method makes it applicable to any domain and where no paired phrase localization annotation is available.
Out-of-body Locomotion: Vectionless Navigation with a Continuous Avatar Representation

Teleportation is a popular and low risk means of navigating in VR. Because teleportation discontinuously translates the user's viewpoint, no optical flow is generated that could lead to vection-induced VR sickness. However, instant viewpoint translations and resulting discontinuous avatar representation is not only detrimental to presence and spatial awareness but also presents a challenge for gameplay design–particularly for multiplayer games. We compare out-of-body locomotion, a hybrid viewpoint technique that lets users seamlessly switch between a first-person and third-person avatar view, to traditional pointer-based teleportation. While in third-person, if the user doesn't move, the camera remains stationary to avoid any optical flow generation. Third-person also lets users precisely and continuously navigate their avatar without risk of getting VR sick. The viewpoint automatically switches back to first-person as soon the users breaks line of sight with their avatar or the user requests to rejoin the avatar with a button press. A user study compares out-of-body locomotion to teleportation with participants (n=22) traversing an obstacle course. Results show that out-of-body locomotion requires significantly fewer (67%) viewpoint transitions than teleportation while there was no significant difference in performance. In addition to being able to offer a continuous avatar representation, participants also deemed out-of-body locomotion to be faster.