Reviewerly/exHarmony huggingface dataset:
Scalable Deep Multimodal Learning for Cross-Modal Retrieval

Cross-modal retrieval takes one type of data as the query to retrieve relevant data of another type. Most of existing cross-modal retrieval approaches were proposed to learn a common subspace in a joint manner, where the data from all modalities have to be involved during the whole training process. For these approaches, the optimal parameters of different modality-specific transformations are dependent on each other and the whole model has to be retrained when handling samples from new modalities. In this paper, we present a novel cross-modal retrieval method, called Scalable Deep Multimodal Learning (SDML). It proposes to predefine a common subspace, in which the between-class variation is maximized while the within-class variation is minimized. Then, it trains m modality-specific networks for m modalities (one network for each modality) to transform the multimodal data into the predefined common subspace to achieve multimodal learning. Unlike many of the existing methods, our method can train different modality-specific networks independently and thus be scalable to the number of modalities. To the best of our knowledge, the proposed SDML could be one of the first works to independently project data of an unfixed number of modalities into a predefined common subspace. Comprehensive experimental results on four widely-used benchmark datasets demonstrate that the proposed method is effective and efficient in multimodal learning and outperforms the state-of-the-art methods in cross-modal retrieval.
Augmented Reality in Design Education: Landscape Architecture Studies as AR Experience

Abstract The rapid and ongoing development of digital technologies continues to create new opportunities for education. Over the last decade this has enabled the establishment of blended learning approaches and online education. More recently, Augmented Reality ( AR ) has emerged as a unique technology that can transform learning experiences across diverse disciplines. This article outlines the development of an AR prototype, Master of Time , which was created to educate first year students and non‐designers on the foundational principles of landscape architecture. This study examines the learning potential and benefits of AR technology with a focus on creating new practices in digital storytelling across situated experiences. In outlining project outcomes, the authors propose a series of critical design principles, strategies and methodologies for educators to apply when developing AR learning experiences across disciplines. Included within this is a framework for transdisciplinary and co‐design collaboration, which is essential for educators working in the forefront of learning technologies.
Learning to Adapt for Stereo

Real world applications of stereo depth estimation require models that are robust to dynamic variations in the environment. Even though deep learning based stereo methods are successful, they often fail to generalize to unseen variations in the environment, making them less suitable for practical applications such as autonomous driving. In this work, we introduce a ``learning-to-adapt'' framework that enables deep stereo methods to continuously adapt to new target domains in an unsupervised manner. Specifically, our approach incorporates the adaptation procedure into the learning objective to obtain a base set of parameters that are better suited for unsupervised online adaptation. To further improve the quality of the adaptation, we learn a confidence measure that effectively masks the errors introduced during the unsupervised adaptation. We evaluate our method on synthetic and real-world stereo datasets and our experiments evidence that learning-to-adapt is, indeed beneficial for online adaptation on vastly different domains.
Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention

We present Vision-based Navigation with Language-based Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to the target objects and thus makes requests by only specifying high-level end-goals, and (b) the agent is capable of sensing when it is lost and querying an advisor, who is more qualified at the task, to obtain language subgoals to make progress. To model language-based assistance, we develop a general framework termed Imitation Learning with Indirect Intervention (I3L), and propose a solution that is effective on the VNLA task. Empirical results show that this approach significantly improves the success rate of the learning agent over other baselines on both seen and unseen environments. Our code and data are publicly available at https://github.com/debadeepta/vnla .
Energy-efficient and quality-aware VM consolidation method

To improve resource utilization and energy efficiency, cloud data centers use virtual machine (VM) consolidation to consolidate VMs to fewer physical machines (PMs) through live VM migration. However, improper VM placement may cause frequent VM migrations and constant on–off switching of PMs, which results in lower service quality and increased energy consumption. In this paper, we address this problem by proposing an effective and efficient VM consolidation approach called EQ-VMC, which has the goal of optimizing energy efficiency and service quality. In our approach, a discrete differential evolution algorithm is developed to search for the global optimum solution for VM placement. By integrating this solution with a set of algorithms proposed for effective host overload detection, VM selection, and under-loaded host detection, EQ-VMC effectively reduces energy consumption and improves quality of service (QoS). Extensive simulation demonstrates its effectiveness and shows its superiority to previous VM consolidation methods.
Pixel and feature level based domain adaptation for object detection in autonomous driving

Annotating large-scale datasets to train modern convolutional neural networks is prohibitively expensive and time-consuming for many real tasks. One alternative is to train the model on labeled synthetic datasets and apply it in the real scenes. However, this straightforward method often fails to generalize well mainly due to the domain bias between the synthetic and real datasets. Many unsupervised domain adaptation (UDA) methods were introduced to address this problem but most of them only focused on the simple classification task. This paper presents a novel UDA model which integrates both image and feature level based adaptations to solve the cross-domain object detection problem. We employ objectives of the generative adversarial network and the cycle consistency loss for image translation. Furthermore, region proposal based feature adversarial training and classification are proposed to further minimize the domain shifts and preserve the semantics of the target objects. Extensive experiments are conducted on several different adaptation scenarios, and the results demonstrate the robustness and superiority of the proposed method.
Feature selection in image analysis: a survey

Image analysis is a prolific field of research which has been broadly studied in the last decades, successfully applied to a great number of disciplines. Since the apparition of Big Data, the number of digital images is explosively growing, and a large amount of multimedia data is publicly available. Not only is it necessary to deal with this increasing number of images, but also to know which features extract from them, and feature selection can help in this scenario. The goal of this paper is to survey the most recent feature selection methods developed and/or applied to image analysis, covering the most popular fields such as image classification, image segmentation, etc. Finally, an experimental evaluation on several popular datasets using well-known feature selection methods is presented, bearing in mind that the aim is not to provide the best feature selection method, but to facilitate comparative studies for the research community.
A New Chaotic S-Box Generation Method Using Parameter Optimization of One Dimensional Chaotic Maps

Chaotic systems have been used to generate the substitutional box structures.The existence of endless possibilities for the selection of the initial conditions and control parameters of the chaotic system has made it a necessity to use the optimization algorithms for generation of chaos based substitutional box structures.Most appropriate initial conditions and control parameters have been determined for four different discrete time chaotic systems using seven different optimization algorithms.A new substitutional box generation algorithm has been proposed using the optimum chaotic parameters values parameter values.
Extractive Text Summarization Using Sentence Ranking

Automatic Text summarization is the technique to identify the most useful and necessary information in a text. It has two approaches 1) Abstractive text summarization and 2) Extractive text summarization. An extractive text summarization means an important information or sentence are extracted from the given text file or original document. In this paper, a novel statistical method to perform an extractive text summarization on single document is demonstrated. The method extraction of sentences, which gives the idea of the input text in a short form, is presented. Sentences are ranked by assigning weights and they are ranked based on their weights. Highly ranked sentences are extracted from the input document so it extracts important sentences which directs to a high-quality summary of the input document and store summary as audio.
Speeding Up Distributed Gradient Descent by Utilizing Non-persistent Stragglers

When gradient descent (GD) is scaled to many parallel computing servers (workers) for large scale machine learning problems, its per-iteration computation time is limited by the straggling workers. Coded distributed GD (DGD) can tolerate straggling workers by assigning redundant computations to the workers, but in most existing schemes, each non-straggling worker transmits one message per iteration to the parameter server (master) after completing all its computations. We allow multiple computations to be conveyed from each worker per iteration in order to exploit computations executed also by the straggling worker. We show that the average completion time per iteration can be reduced significantly at a reasonable increase in the communication load. We also propose a general coded DGD technique which can trade-off the average computation time with the communication load.
Data Efficient Unsupervised Domain Adaptation For Cross-modality Image Segmentation

Deep learning models trained on medical images from a source domain ( $$ e.g. $$ imaging modality) often fail when deployed on images from a different target domain, despite imaging common anatomical structures. Deep unsupervised domain adaptation (UDA) aims to improve the performance of a deep neural network model on a target domain, using solely unlabelled target domain data and labelled source domain data. However, current state-of-the-art methods exhibit reduced performance when target data is scarce. In this work, we introduce a new data efficient UDA method for multi-domain medical image segmentation. The proposed method combines a novel VAE-based feature prior matching, which is data-efficient, and domain adversarial training to learn a shared domain-invariant latent space which is exploited during segmentation. Our method is evaluated on a public multi-modality cardiac image segmentation dataset by adapting from the labelled source domain (3D MRI) to the unlabelled target domain (3D CT). We show that by using only one single unlabelled 3D CT scan, the proposed architecture outperforms the state-of-the-art in the same setting. Finally, we perform ablation studies on prior matching and domain adversarial training to shed light on the theoretical grounding of the proposed method.
HPILN: a feature learning framework for cross‐modality person re‐identification

Most video surveillance systems use both RGB and infrared cameras, making it a vital technique to re-identify a person cross the RGB and infrared modalities. This task can be challenging due to both the cross-modality variations caused by heterogeneous images in RGB and infrared, and the intra-modality variations caused by the heterogeneous human poses, camera position, light brightness etc. To meet these challenges, a novel feature learning framework, hard pentaplet and identity loss network (HPILN), is proposed. In the framework existing single-modality re-identification models are modified to fit for the cross-modality scenario, following which specifically designed hard pentaplet loss and identity loss are used to increase the accuracy of the modified cross-modality re-identification models. Based on the benchmark of the SYSU-MM01 dataset, extensive experiments have been conducted, showing that the authors’ method outperforms all existing ones in terms of cumulative match characteristic curve and mean average precision.
Deep Incremental Hashing Network for Efficient Image Retrieval

Hashing has shown great potential in large-scale image retrieval due to its storage and computation efficiency, especially the recent deep supervised hashing methods. To achieve promising performance, deep supervised hashing methods require a large amount of training data from different classes. However, when images of new categories emerge, existing deep hashing methods have to retrain the CNN model and generate hash codes for all the database images again, which is impractical for large-scale retrieval system. In this paper, we propose a novel deep hashing framework, called Deep Incremental Hashing Network (DIHN), for learning hash codes in an incremental manner. DIHN learns the hash codes for the new coming images directly, while keeping the old ones unchanged. Simultaneously, a deep hash function for query set is learned by preserving the similarities between training points. Extensive experiments on two widely used image retrieval benchmarks demonstrate that the proposed DIHN framework can significantly decrease the training time while keeping the state-of-the-art retrieval accuracy.
Design of a SWOT Analysis Model and its Evaluation in Diverse Digital Business Ecosystem Contexts

Investigating the Strengths, Weaknesses, Opportunities and Threats (SWOT) of enterprise systems is popular among business researchers in major organizations. Many establishments carry out SWOT analysis at strategic planning, quality control while formulating government policies and legislations. In the digital ecosystems scenarios, the SWOT activities need a great deal of attention, in particular, while designing and promoting new strategies of multiple industry scenarios in the Integrated Project Management context, keeping in view complex business operations. Information solutions may not have choices, failing to address priorities and provide alternate solutions. We focus on digital ecosystem methodologies, in which the business and organizational issues, challenges and priorities are addressed. The purpose of the research is designing a new SWOT model in which the elements are modelled to interrogate managers’ views to oversee new insights of a variety of business contexts that can guide SWOT analyzers and provide digital ecosystem services in multiple industry operations in an optimum manner. Issues and challenges of elements of SWOT of several public and private sector companies are analyzed, documented and modelled to evaluate unified metadata representing multiple industry views, their visualization and interpretation in new knowledge domains.
Unsupervised Learning of Action Classes With Continuous Temporal Embedding

The task of temporally detecting and segmenting actions in untrimmed videos has seen an increased attention recently. One problem in this context arises from the need to define and label action boundaries to create annotations for training which is very time and cost intensive. To address this issue, we propose an unsupervised approach for learning action classes from untrimmed video sequences. To this end, we use a continuous temporal embedding of framewise features to benefit from the sequential nature of activities. Based on the latent space created by the embedding, we identify clusters of temporal segments across all videos that correspond to semantic meaningful action classes. The approach is evaluated on three challenging datasets, namely the Breakfast dataset, YouTube Instructions, and the 50Salads dataset. While previous works assumed that the videos contain the same high level activity, we furthermore show that the proposed approach can also be applied to a more general setting where the content of the videos is unknown.
Explicit Bias Discovery in Visual Question Answering Models

Researchers have observed that Visual Question Answering (VQA ) models tend to answer questions by learning statistical biases in the data. For example, their answer to the question “What is the color of the grass?” is usually “Green”, whereas a question like “What is the title of the book?” cannot be answered by inferring statistical biases. It is of interest to the community to explicitly discover such biases, both for understanding the behavior of such models, and towards debugging them. Our work address this problem. In a database, we store the words of the question, answer and visual words corresponding to regions of interest in attention maps. By running simple rule mining algorithms on this database, we discover human-interpretable rules which give us unique insight into the behavior of such models. Our results also show examples of unusual behaviors learned by models in attempting VQA tasks.
VoltJockey

ARM TrustZone builds a trusted execution environment based on the concept of hardware separation. It has been quite successful in defending against various software attacks and forcing attackers to explore vulnerabilities in interface designs and side channels. The recently reported CLKscrew attack breaks TrustZone through software by overclocking CPU to generate hardware faults. However, overclocking makes the processor run at a very high frequency, which is relatively easy to detect and prevent, for example by hardware frequency locking. In this paper, we propose an innovative software-controlled hardware fault-based attack, VoltJockey, on multi-core processors that adopt dynamic voltage and frequency scaling (DVFS) techniques for energy efficiency. Unlike CLKscrew, we manipulate the voltages rather than the frequencies via DVFS unit to generate hardware faults on the victim cores, which makes VoltJockey stealthier and harder to prevent than CLKscrew. We deliberately control the fault generation to facilitate differential fault analysis to break TrustZone. The entire attack process is based on software without any involvement of hardware. We implement VoltJockey on an ARM-based Krait processor from a commodity Android phone and demonstrate how to reveal the AES key from TrustZone and how to breach the RSA-based TrustZone authentication. These results suggest that VoltJockey has a comparable efficiency to side channels in obtaining TrustZone-guarded credentials, as well as the potential of bypassing the RSA-based verification to load untrusted applications into TrustZone. We also discuss both hardware-based and software-based countermeasures and their limitations.
Grounded Human-Object Interaction Hotspots From Video

Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements. We propose an approach to learn human-object interaction "hotspots" directly from video. Rather than treat affordances as a manually supervised semantic segmentation task, our approach learns about interactions by watching videos of real human behavior and anticipating afforded actions. Given a novel image or video, our model infers a spatial hotspot map indicating where an object would be manipulated in a potential interaction even if the object is currently at rest. Through results with both first and third person video, we show the value of grounding affordances in real human-object interactions. Not only are our weakly supervised hotspots competitive with strongly supervised affordance methods, but they can also anticipate object interaction for novel object categories. Project page: http://vision.cs.utexas.edu/projects/interaction-hotspots/.
Semi-Supervised Video Salient Object Detection Using Pseudo-Labels

Deep learning-based video salient object detection has recently achieved great success with its performance significantly outperforming any other unsupervised methods. However, existing data-driven approaches heavily rely on a large quantity of pixel-wise annotated video frames to deliver such promising results. In this paper, we address the semi-supervised video salient object detection task using pseudo-labels. Specifically, we present an effective video saliency detector that consists of a spatial refinement network and a spatiotemporal module. Based on the same refinement network and motion information in terms of optical flow, we further propose a novel method for generating pixel-level pseudo-labels from sparsely annotated frames. By utilizing the generated pseudo-labels together with a part of manual annotations, our video saliency detector learns spatial and temporal cues for both contrast inference and coherence enhancement, thus producing accurate saliency maps. Experimental results demonstrate that our proposed semi-supervised method even greatly outperforms all the state-of-the-art fully supervised methods across three public benchmarks of VOS, DAVIS, and FBMS.
Aggregation via Separation: Boosting Facial Landmark Detector With Semi-Supervised Style Translation

Facial landmark detection, or face alignment, is a fundamental task that has been extensively studied. In this paper, we investigate a new perspective of facial landmark detection and demonstrate it leads to further notable improvement. Given that any face images can be factored into space of style that captures lighting, texture and image environment, and a style-invariant structure space, our key idea is to leverage disentangled style and shape space of each individual to augment existing structures via style translation. With these augmented synthetic samples, our semi-supervised model surprisingly outperforms the fully-supervised one by a large margin. Extensive experiments verify the effectiveness of our idea with state-of-the-art results on WFLW, 300W, COFW, and AFLW datasets. Our proposed structure is general and could be assembled into any face alignment frameworks. The code is made publicly available at https://github.com/thesouthfrog/stylealign.