Reviewerly/exHarmony huggingface dataset:
Optimal Ensemble Scheme for Human Activity Recognition and Floor Detection Based on AutoML and Weighted Soft Voting Using Smartphone Sensors

In this article, human activity recognition (HAR) attempts to recognize the activities of an object in a multistory building from data retrieved via smartphone-based sensors (SBSs). Most publications based on machine learning (ML) report the development of a suitable architecture to improve classification accuracy by increasing the parameters of the architecture pertaining to HAR. Due to robust and automated ML, it is quite possible to develop a versatile approach to improve the accuracy of HAR. This research proposes an optimal ensemble HAR and floor detection (OEC-HAFD) scheme based on automated learning (AutoML) and weighted soft voting (WSF) using SBS to improve the recognition rate. The proposed HAR-FD scheme is developed based on twofold mechanisms. First, an AutoML paradigm is employed to find optimal supervised models based on performance. Second, top-ranked (optimal) models are combined using the WSF mechanism to classify HAR on various floors of a multistory building. The proposed scheme is developed based on real-time SBS: accelerometer and barometer data. The accelerometer data are used to detect activity by observing the magnitude of the sensor measurements. Similarly, the barometer sensor detects floor height by using pressure and altitude data. Furthermore, we analyze the performance of the proposed optimal ensemble classifier (OEC) and compare them with the state-of-the-art classifiers. Based on the result analysis, it is clearly seen that the proposed OEC outperforms the performance of the individual classifiers. Moreover, our proposed HAR-FD can be leveraged as a robust solution to accurately recognize human activities in multistory buildings compared to the existing standalone models.
Leveraging Equivariant Features for Absolute Pose Regression

While end-to-end approaches have achieved state-of-the-art performance in many perception tasks, they are not yet able to compete with 3D geometry-based methods in pose estimation. Moreover, absolute pose regression has been shown to be more related to image retrieval. As a result, we hypothesize that the statistical features learned by classical Convolutional Neural Networks do not carry enough geometric information to reliably solve this inherently geometric task. In this paper, we demonstrate how a translation and rotation equivariant Convolutional Neural Network directly induces representations of camera motions into the feature space. We then show that this geometric property allows for implicitly augmenting the training data under a whole group of image plane-preserving transformations. Therefore, we argue that directly learning equivariant features is preferable than learning data-intensive intermediate representations. Comprehensive experimental validation demonstrates that our lightweight model outperforms existing ones on standard datasets. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
HyperDet3D: Learning a Scene-conditioned 3D Object Detector

A bathtub in a library, a sink in an office, a bed in a laundry room - the counter-intuition suggests that scene provides important prior knowledge for 3D object detection, which instructs to eliminate the ambiguous detection of similar objects. In this paper, we propose HyperDet3D to explore scene-conditioned prior knowledge for 3D object detection. Existing methods strive for better representation of local elements and their relations without scene-conditioned knowledge, which may cause ambiguity merely based on the understanding of individual points and object candidates. Instead, HyperDet3D simultaneously learns scene-agnostic embeddings and scene-specific knowledge through scene-conditioned hypernetworks. More specifically, our HyperDet3D not only explores the sharable abstracts from various 3D scenes, but also adapts the detector to the given scene at test time. We propose a discriminative Multi-head Scene-specific Attention (MSA) module to dynamically control the layer parameters of the detector conditioned on the fusion of scene-conditioned knowledge. Our HyperDet3D achieves state-of-the-art results on the 3D object detection benchmark of the ScanNet and SUN RGB-D datasets. Moreover, through cross-dataset evaluation, we show the acquired scene-conditioned prior knowledge still takes effect when facing 3D scenes with domain gap.
FAR Planner: Fast, Attemptable Route Planner using Dynamic Visibility Update

Path planning in unknown environments remains a challenging problem, as the environment is gradually observed during the navigation, the underlying planner has to update the environment representation and replan, promptly and constantly, to account for the new observations. In this paper, we present a visibility graph-based planning framework capable of dealing with navigation tasks in both known and unknown environments. The planner employs a polygonal representation of the environment and constructs the representation by extracting edge points around obstacles to form enclosed polygons. With that, the method dynamically updates a global visibility graph using a two-layered data structure, expanding the visibility edges along with the navigation, and removing edges that become occluded by newly observed obstacles. When navigating in unknown environments, the method is attemptable in discovering a way to the goal by picking up the environment layout on the fly, updating the visibility graph, and fast replanning corresponding to the newly observed environment. We evaluate the method in simulated and real-world settings. The method shows the capability to attempt and navigate through unknown environments, reducing travel time by up to 12-47% from search-based methods: A*, D* Lite, and more than 24-35% from sampling-based methods: RRT*, BIT*, and SPARS.
NinjaDesc: Content-Concealing Visual Descriptors via Adversarial Learning

In the light of recent analyses on privacy-concerning scene revelation from visual descriptors, we develop descriptors that conceal the input image content. In particular, we propose an adversarial learning framework for training visual descriptors that prevent image reconstruction, while maintaining the matching accuracy. We let a feature encoding network and image reconstruction network compete with each other, such that the feature encoder tries to impede the image reconstruction with its generated descriptors, while the reconstructor tries to recover the input image from the descriptors. The experimental results demonstrate that the visual descriptors obtained with our method significantly deteriorate the image reconstruction quality with minimal impact on correspondence matching and camera localization performance.
PIE-Net: Photometric Invariant Edge Guided Network for Intrinsic Image Decomposition

Intrinsic image decomposition is the process of recovering the image formation components (reflectance and shading) from an image. Previous methods employ either explicit priors to constrain the problem or implicit constraints as formulated by their losses (deep learning). These methods can be negatively influenced by strong illumination conditions causing shading-reflectance leakages. Therefore, in this paper, an end-to-end edge-driven hybrid CNN approach is proposed for intrinsic image decomposition. Edges correspond to illumination invariant gradients. To handle hard negative illumination transitions, a hierarchical approach is taken including global and local refinement layers. We make use of attention layers to further strengthen the learning process. An extensive ablation study and large scale experiments are conducted showing that it is beneficial for edge-driven hybrid IID networks to make use of illumination invariant descriptors and that separating global and local cues helps in improving the performance of the network. Finally, it is shown that the proposed method obtains state of the art performance and is able to generalise well to real world images. The project page with pretrained models, finetuned models and network code can be found at https://ivi.fnwi.uva.nl/cv/pienet/.
Location-Free Human Pose Estimation

Human pose estimation (HPE) usually requires large-scale training data to reach high performance. However, it is rather time-consuming to collect high-quality and fine-grained annotations for human body. To alleviate this issue, we revisit HPE and propose a location-free framework without supervision of keypoint locations. We reformulate the regression-based HPE from the perspective of classification. Inspired by the CAM-based weakly-supervised object localization, we observe that the coarse keypoint locations can be acquired through the part-aware CAMs but unsatisfactory due to the gap between the fine-grained HPE and the object-level localization. To this end, we propose a customized transformer framework to mine the fine-grained representation of human context, equipped with the structural relation to capture subtle differences among keypoints. Concretely, we design a Multi-scale Spatial-guided Context Encoder to fully capture the global human context while focusing on the part-aware regions and a Relation-encoded Pose Prototype Generation module to encode the structural relations. All these works together for strengthening the weak supervision from image-level category labels on locations. Our model achieves competitive performance on three datasets when only supervised at a category-level and importantly, it can achieve comparable results with fully-supervised methods with only 25% location labels on MS-COCO and MPII.
Hierarchical Feature Embedding for Visual Tracking

Features extracted by existing tracking methods may contain instance- and category-level information. However, it usually occurs that either instance- or category-level information uncontrollably dominates the feature embeddings depending on the training data distribution, since the two types of information are not explicitly modeled. A more favorable way is to produce features that emphasize both types of information in visual tracking. To achieve this, we propose a hierarchical feature embedding model which separately learns the instance and category information, and progressively embeds them. We develop the instance-aware and category-aware modules that collaborate from different semantic levels to produce discriminative and robust feature embeddings. The instance-aware module concentrates on the instance level in which the inter-video contrastive learning mechanism is adopted to facilitate inter-instance separability and intra-instance compactness. However, it is challenging to force the intra-instance compactness by using instance-level information alone because of the prevailing appearance changes of the instance in visual tracking. To tackle this problem, the category-aware module is employed to summarize high-level category information which remains robust despite instance-level appearance changes. As such, intra-instance compactness can be effectively improved by jointly leveraging the instance- and category-aware modules. Experimental results on various benchmarks demonstrate the proposed method performs favorably against the state-of-the-arts. The code is available on https://github.com/zxgravity/CIA .
SEEMQTT: Secure End-to-End MQTT-Based Communication for Mobile IoT Systems Using Secret Sharing and Trust Delegation

The publish/subscribe (Pub/Sub) model offers a communication scheme that is appropriate for a variety of mobile Internet of Things (IoT) systems (e.g., autonomous vehicles). In most of these systems, ensuring the end-to-end (E2E) security of exchanged information is a critical requirement. However, the Pub/Sub scheme lacks appropriate mechanisms to ensure the E2E security, even when state-of-the-art solutions, such as transport layer security (TLS) or attribute-based encryption (ABE), were adopted. These solutions either do not offer E2E security or are infeasible to be adopted in mobile IoT systems with resource-constrained platforms. In this article, we propose a framework, so-called SEEMQTT, to ensure secure E2E Pub/Sub-based communication for mobile IoT systems. Our solution allows the publisher to encrypt the published messages and control which subscribers can decrypt these messages without violating the decoupling requirement of the Pub/Sub model. Our solution leverages multiple honest-but-curious KeyStores to store secret shares generated from a secret key using a secret sharing scheme. The links between the publisher and every KeyStores are secured using identity-based encryption (IBE). The publisher uses the secret key to encrypt published messages. Trust delegation is used to authorize certain subscribers to access these shares and consequently decrypt the published messages. We provide an Arduino-based library that implements our proposed protocol. Also, we perform an extensive performance evaluation using real IoT hardware. Experimental results show that adopting our proposed solution, SEEMQTT, makes E2E security for mobile IoT systems feasible.
Enhancing Adversarial Robustness for Deep Metric Learning

Owing to security implications of adversarial vulnerability, adversarial robustness of deep metric learning models has to be improved. In order to avoid model collapse due to excessively hard examples, the existing defenses dismiss the min-max adversarial training, but instead learn from a weak adversary inefficiently. Conversely, we propose Hardness Manipulation to efficiently perturb the training triplet till a specified level of hardness for adversarial training, according to a harder benign triplet or a pseudo-hardness function. It is flexible since regular training and min-max adversarial training are its boundary cases. Besides, Gradual Adversary, a family of pseudo-hardness functions is proposed to gradually increase the specified hardness level during training for a better balance between performance and robustness. Additionally, an Intra-Class Structure loss term among benign and adversarial examples further improves model robust-ness and efficiency. Comprehensive experimental results suggest that the proposed method, although simple in its form, overwhelmingly outperforms the state-of-the-art de-fenses in terms of robustness, training efficiency, as well as performance on benign examples.
Medical Named Entity Recognition using Surrounding Sequences Matching

Since the development of information technologies, there is a huge amount of electronic documents that was written by medical specialists and are rich of useful information needed to make critical decisions in several medical tasks. Thus, a doctor must have a big knowledge and he is responsible for every decision he takes for patients. In fact, the doctor should read, with full concentration, many electronic narrative documents to collect the necessary information. Unfortunately, it's too tiring to read all necessary information about drugs, diseases and patient due to the large amount of documents that are increasing every day. Consequently, so many medical errors can happen and even can cause fatalities. On the other hand, information extraction is such a good field that can handle this problem. One of the most important main task in this field is the Named Entity Recognition (NER) and its role was to identify the medical named entities, such as drug, disease or treatment, from an unstructured text written in natural language. However, in order to treat the narrative text, natural language tasks should be performed before NER. In our paper, we introduce a named entity recognition method, called NESSMa (Named Entity tagging by Surrounding Sequence Matching), and based on sequence tagging, it is able to annotate the words of a sentence using Bidirectional Long Short-Term Memory neural network with Conditional Random Field (BiLSTM-CRF) model. We pass the Bidirectional Encoder Representations from Transformers (BERT) word embedding as feature together with the Part of Speech (PoS) of the word and the cue sequence information. The cue sequence information indicates if a word belongs to a named entity surrounding sequence based on word edit distance. For that, we have automatically constructed a dictionary of named entities’ surrounding sequences for each entity type using a train set. As expected, experiments shows that adding the cue sequence information is able to improve the results according to F1-measure and outperform state-of-the-art methods.
Faster VoxelPose: Real-time 3D Human Pose Estimation by Orthographic Projection

While the voxel-based methods have achieved promising results for multi-person 3D pose estimation from multi-cameras, they suffer from heavy computation burdens, especially for large scenes. We present Faster VoxelPose to address the challenge by re-projecting the feature volume to the three two-dimensional coordinate planes and estimating X, Y, Z coordinates from them separately. To that end, we first localize each person by a 3D bounding box by estimating a 2D box and its height based on the volume features projected to the xy-plane and z-axis, respectively. Then for each person, we estimate partial joint coordinates from the three coordinate planes separately which are then fused to obtain the final 3D pose. The method is free from costly 3D-CNNs and improves the speed of VoxelPose by ten times and meanwhile achieves competitive accuracy as the state-of-the-art methods, proving its potential in real-time applications.
From linguistic features to their extractions: Understanding the semantics of a concept

Understanding the meaning of a word has always been a challenging task for machines. There are circumstances when even an easy word becomes difficult to understand. This understanding is not limited to its meaning but other criteria like identifying similar words, solving ambiguity, co reference resolution, etc. Semantic features and semantic relations can be identified as the building blocks for the semantic illustration of a concept. Understanding a language by machine requires understanding of words both linguistically and computationally. This paper tries to critically review the semantic features and relations created with their extraction methods.
VecGAN: Image-to-Image Translation with Interpretable Latent Directions

We propose VecGAN, an image-to-image translation framework for facial attribute editing with interpretable latent directions. Facial attribute editing task faces the challenges of precise attribute editing with controllable strength and preservation of the other attributes of an image. For this goal, we design the attribute editing by latent space factorization and for each attribute, we learn a linear direction that is orthogonal to the others. The other component is the controllable strength of the change, a scalar value. In our framework, this scalar can be either sampled or encoded from a reference image by projection. Our work is inspired by the latent space factorization works of fixed pretrained GANs. However, while those models cannot be trained end-to-end and struggle to edit encoded images precisely, VecGAN is end-to-end trained for image translation task and successful at editing an attribute while preserving the others. Our extensive experiments show that VecGAN achieves significant improvements over state-of-the-arts for both local and global edits.
Trajectory Synthesis for a UAV Swarm Based on Resilient Data Collection Objectives

The use of Unmanned Aerial Vehicles (UAVs) for collecting data from remotely located sensor systems is emerging. The data can be time-sensitive and require to be transmitted to a data processing center. However, planning the trajectory for a swarm of UAVs depends on multi-fold constraints, such as data collection requirements, UAV maneuvering capacities, and budget limitations. Since a UAV may fail or be compromised, it is important to provide necessary resilience to such contingencies, thus ensuring data security. It is important to provide the UAVs with efficient spatio-temporal trajectories so that they can efficiently cover necessary data sources. In this work, we present Synth4UAV, a formal approach for automated synthesis of efficient trajectories for a UAV swarm by logically modeling the aerial space and data point topology, UAV moves, and associated constraints in terms of the turning and climbing angle, fuel usage, data collection point coverage, data freshness, and resiliency properties. We use efficient, logical formulas to encode and solve the complex model. The solution to the model provides the routing and maneuvering plan for each UAV, including the time to visit the points on the paths and corresponding fuel usage such that the necessary data points are visited while satisfying the resiliency requirements. We evaluate the proposed trajectory synthesizer, and the results show that the relationship among different parameters follows the requirements while the tool scales well with the problem size.
Learning Audio-Video Modalities from Image Captions

There has been a recent explosion of large-scale image-text datasets, as images with alt-text captions can be easily obtained online. Obtaining large-scale, high quality data for video in the form of text-video and text-audio pairs however, is more challenging. To close this gap we propose a new video mining pipeline which involves transferring captions from image captioning datasets to video clips with no additional manual effort. Using this pipeline, we create a new large-scale, weakly labelled audio-video captioning dataset consisting of millions of paired clips and captions. We show that training a multimodal transformer based model on this data achieves competitive performance on video retrieval and video captioning, matching or even outperforming HowTo100M pretraining with 20x fewer clips. We also show that our mined clips are suitable for text-audio pretraining, and achieve state of the art results for the task of audio retrieval.
Approach of a Multilevel Secret Sharing Scheme for Extracted Text Data

Text data pre-processing has become essential in research fields like Information Retrieval (IR), Natural Language Processing (NLP), and text mining. It extracts valuable and nontrivial information from unstructured text data. Text data plays an essential role in the digital era as various types of information and data are being generated in an unstructured way. In this work, Multi-level Secret Sharing Scheme (MSSS) based novel approach has been proposed to ensure the security of the most informative and sensitive text data extracted through text preprocessing steps from the text document. The proposed model employs tokenization, stop word removal, and stemming as preprocessing techniques, which help examine text documents. Three types (electronic mail, whats-app messages, and text messages) of datasets have been contributed to this work for performing the experiments, where the model achieves 100% correlation between original and reconstructed text.