Miracle-dz/UniFAR-FTUs huggingface dataset:
DeepVoting: An Explainable Framework for Semantic Part Detection under Partial Occlusion

In this paper, we study the task of detecting semantic parts of an object. This is very important in computer vision, as it provides the possibility to parse an object as human do, and helps us better understand object detection algorithms. Also, detecting semantic parts is very challenging especially when the parts are partially or fully occluded. In this scenario, the popular proposal-based methods like Faster-RCNN often produce unsatisfactory results, because both the proposal extraction and classification stages may be confused by the irrelevant occluders. To this end, we propose a novel detection framework, named DeepVoting, which accumulates local visual cues, called visual concepts (VC), to locate the semantic parts. Our approach involves adding two layers after the intermediate outputs of a deep neural network. The first layer is used to extract VC responses, and the second layer performs a voting mechanism to capture the spatial relationship between VC's and semantic parts. The benefit is that each semantic part is supported by multiple VC's. Even if some of the supporting VC's are missing due to occlusion, we can still infer the presence of the target semantic part using the remaining ones. To avoid generating an exponentially large training set to cover all occlusion cases, we train our model without seeing occlusion and transfer the learned knowledge to deal with occlusions. This setting favors learning the models which are naturally robust and adaptive to occlusions instead of over-fitting the occlusion patterns in the training data. In experiments, DeepVoting shows significantly better performance on semantic part detection in occlusion scenarios, compared with Faster-RCNN, with one order of magnitude fewer parameters and 2.5x testing speed. In addition, DeepVoting is explainable as the detection result can be diagnosed via looking up the voted VC's.
On-Chip CNN Accelerator for Image Super-Resolution

To implement convolutional neural networks (CNN) in hardware, the state-of-the-art CNN accelerators pipeline computation and data transfer stages using an off-chip memory and simultaneously execute them on the same timeline. However, since a large amount of feature maps generated during the operation should be transmitted to the off-chip memory, the pipeline stage length is determined by the off-chip data transfer stage. Fusion architectures that can fuse multiple layers have been proposed to solve this problem, but applications such as super-resolution (SR) require a large amount of an on-chip memory because of the high resolution of the feature maps. In this paper, we propose a novel on-chip CNN accelerator for SR to optimize the CNN dataflow in the on-chip memory. First, the convolution loop optimization technique is proposed to prevent using a frame buffer. Second, we develop a combined convolutional layer processor to reduce the buffer size used to store the feature maps. Third, we explore how to perform low-cost multiply-and-accumulate operations in the deconvolutional layer used in SR. Finally, we propose a two-stage quantization algorithm to select the optimized hardware size for the limited number of DSPs to implement the on-chip CNN accelerator. We evaluate our proposed accelerator with FSRCNN, which is most popular as the CNN-based SR algorithm. Experimental results show that the proposed accelerator requires 9.21ms to achieve an output image with the 2560x1440 pixel resolution, which is 36 times faster than the conventional method. In addition, we reduce the on-chip memory usage and DSP usage by 4 times and 1.44 times, respectively, compared to the conventional methods.
Learning a Semantically Discriminative Joint Space for Attribute Based Person Re-identification

Attribute based person re-identification (Re-ID) aims to search persons in large-scale image databases using attribute queries. Despite the great practical significance,the huge gap between semantic attributes and visual images makes it a very challenging task. Existing researches usually focus on the match between query attributes and attributes scores predicted from human images, which suffers from imperfect attribute prediction and low discriminability. In this work, we propose to formulate the attribute based person Re-ID as a joint space learning problem. To alleviate the negative impact resulted by the huge heterogeneous gap between different modalities, we apply a novel adversary training strategy to generate homogeneous features for both modalities, providing distribution alignment between different modalities in the feature level and keeping semantic consistency across modalities. Our experiments validate the effectiveness of our model, and show great improvement on the performance over the state of the art models.
Evolutionary generation of adversarial examples for deep and shallow machine learning models

Studying vulnerability of machine learning models to adversarial examples is an important way to understand their robustness and generalization properties. In this paper, we propose a genetic algorithm for generating adversarial examples for machine learning models. Such approach is able to find adversarial examples without the access to model's parameters. Different models are tested, including both deep and shallow neural networks architectures. We show that RBF networks and SVMs with RBF kernels tend to be rather robust and not prone to misclassification of adversarial examples.
Improving the Generalizability of Deep Neural Network Based Speech Enhancement

Enhancing noisy speech is an important task to restore its quality and to improve its intelligibility. In traditional non-machine-learning (ML) based approaches the parameters required for noise reduction are estimated blindly from the noisy observation while the actual filter functions are derived analytically based on statistical assumptions. Even though such approaches generalize well to many different acoustic conditions, the noise suppression capability in transient noises is low. More recently, ML and especially deep learning has been employed for speech enhancement and studies show promising results in noise types where non-ML based approaches fail. However, due to their data-driven nature, the generalizability of ML based approaches to unknown noise types is still discussed. To improve the generalizability of ML based algorithms and to enhance the noise suppression of non-ML based methods, we propose a combination of both approaches. For this, we employ the a priori signal-to-noise ratio (SNR) and the a posteriori SNR estimated by non-ML based algorithms as input features in a deep neural network (DNN) based enhancement scheme. We show that this approach allows ML based speech estimators to generalize quickly to unknown noise types even if only few noise conditions have been seen during training. Instrumental measures such as Perceptual Evaluation of Speech Quality (PESQ) and the segmental SNR indicate strong improvements in unseen conditions when using the proposed features. Listening experiments clearly confirm the improved generalization of our proposed combination.
Minimum Information Exchange in Distributed Systems

Efficient representation of observed data has many applications in various domains of engineering and science. Having a good representation of data, downstream tasks such as classification, regression, and decision making become easier. Even though representing still datasets such as images is a living branch in machine learning, representation of dynamical systems has received less attention. One area where such representation is beneficial is distributed control systems. Distributed systems can be modeled as graphs consisting of nodes that exchange information. This exchange is often costly (e.g. when happening over a shared network) and thus needs to be minimized, while a predefined objective is still satisfied. In this work, we present a method to represent a dynamical system efficiently as a combination of state and local approximate model to fulfill a criterion similar to Minimum Description Length (MDL) principle. MDL is used in machine learning and statistics to state that the best explanation for data is one which yields the best compression. We empirically show the efficacy of the method for several dynamical systems and justify the logic behind the proposed criterion for model selection. Moreover, the method is applied to event-triggered state estimation, used in monitoring and control, and shown to yield a unified framework of previous approaches.
Particle Filter Networks: End-to-End Probabilistic Localization From Visual Observations

Particle filters sequentially approximate posterior distributions by sampling representative points and updating them independently. The idea is applied in various domains, e.g. reasoning with uncertainty in robotics. A remaining challenge is constructing probabilistic models of the system, which can be especially hard for complex sensors, e.g. a camera. We introduce the Particle Filter Networks (PF-nets) that encode both a learned probabilistic system model and the particle filter algorithm in a single neural network architecture. The unified representation allows learning models end-to-end, circumventing the difficulties of conventional model-based methods. We applied PF-nets to a challenging visual localization task that requires matching visual features from camera images with the geometry encoded in a 2-D floor map. In preliminary experiments end-to-end PF-nets consistently outperformed alternative learning architectures, as well as conventional model-based methods.
Deep Image Smoothing based on Texture and Structure Guidance

Image smoothing is a fundamental task in computer vision, which aims to retain salient structures and remove insignificant textures. In this paper, we tackle the natural deficiency of existing methods, that they cannot properly distinguish textures and structures with similar low-level appearance. While deep learning approaches have addressed preserving structures, they do not yet properly address textures. To this end, we build a texture prediction network (TPN) that learns from a various of natural textures. We then combine this with a structure prediction network (SPN) so that the final double-guided filtering network (DGFN) is informed where are the textures to remove ("texture-awareness") and where are the structures to preserve ("structure-awareness"). The proposed model is easy to implement and shows excellent performance on real images in the wild as well as our synthetic dataset.
On-device Scalable Image-based Localization

We present the scalable design of an entire on-device system for large-scale urban localization. The proposed design integrates compact image retrieval and 2D-3D correspondence search to estimate the camera pose in a city region of extensive coverage. Our design is GPS agnostic and does not require the network connection. The system explores the use of an abundant dataset: Google Street View (GSV). In order to overcome the resource constraints of mobile devices, we carefully optimize the system design at every stage: we use state-of-the-art image retrieval to quickly locate candidate regions and limit candidate 3D points; we propose a new hashing-based approach for fast computation of 2D-3D correspondences and new one-many RANSAC for accurate pose estimation. The experiments are conducted on benchmark datasets for 2D-3D correspondence search and on a database of over 227K Google Street View (GSV) images for the overall system. Results show that our 2D-3D correspondence search achieves state-of-the-art performance on some benchmark datasets and our system can accurately and quickly localize mobile images; the median error is less than 4 meters and the processing time is averagely less than 10s on a typical mobile device.
Personalized Attention-Aware Exposure Control Using Reinforcement Learning

We propose a reinforcement learning approach for real-time exposure control of a mobile camera that is personalizable. Our approach is based on Markov Decision Process (MDP). In the camera viewfinder or live preview mode, given the current frame, our system predicts the change in exposure so as to optimize the trade-off among image quality, fast convergence, and minimal temporal oscillation. We model the exposure prediction function as a fully convolutional neural network that can be trained through Gaussian policy gradient in an end-to-end fashion. As a result, our system can associate scene semantics with exposure values; it can also be extended to personalize the exposure adjustments for a user and device. We improve the learning performance by incorporating an attention module that links semantics with exposure. This attention module generalizes the conventional spot or matrix metering techniques. We validate our system using the MIT FiveK and our own datasets captured using iPhone 7 and Google Pixel. Experimental results show that our system exhibits stable real-time behavior while improving visual quality compared to what is achieved through native camera control.
Stacked Semantic-Guided Attention Model for Fine-Grained Zero-Shot Learning

Zero-Shot Learning (ZSL) is achieved via aligning the semantic relationships between the global image feature vector and the corresponding class semantic descriptions. However, using the global features to represent fine-grained images may lead to sub-optimal results since they neglect the discriminative differences of local regions. Besides, different regions contain distinct discriminative information. The important regions should contribute more to the prediction. To this end, we propose a novel stacked semantics-guided attention (S2GA) model to obtain semantic relevant features by using individual class semantic features to progressively guide the visual features to generate an attention map for weighting the importance of different local regions. Feeding both the integrated visual features and the class semantic features into a multi-class classification architecture, the proposed framework can be trained end-to-end. Extensive experimental results on CUB and NABird datasets show that the proposed approach has a consistent improvement on both fine-grained zero-shot classification and retrieval tasks.
A Study of Biometric Feature for a Recall-Based Behavioral Graphical Mobile Authentication

With the increasing use of mobile devices, including smart phones and tablets, in today's IT infrastructure, the paradigms of computer applications have been significantly changed. One area of studies is to utilize a new authentication scheme that will provide a better protection on access control and privacy of the mobile device user. Currently, most of the mobile devices have implemented scheme such as biometric or graphical user authentication. However, study shows people still comfortably use password code as the primary mean for authentication, even it is proven unsecure. In this paper, we propose a graphical mobile authentication scheme that integrates the psychologically popular passcode into a behavioral biometric authentication. This authentication scheme provides an easy transition from a simple passcode to a more secure recall-based graphical mode and helps to prevent shoulder surfing that many graphical authentications suffered from.
Optimizing Caching Policy at Base Stations by Exploiting User Preference and Spatial Locality

Most prior works of proactive caching at wireless edge optimize caching policies under the following assumptions: the preference of each user is identical to content popularity, all users request contents with the same active level and at uniformly-distributed locations. In this paper, we investigate what happens without these assumptions. To this end, we establish a framework to optimize caching policy at base stations exploiting user preference, active level, and spatial locality. We obtain optimal caching policy to minimize the weighted sum of the file download time averaged over all file requests and user locations in the network (reflecting network performance) and the maximal weighted download time averaged over possible file requests and locations of each user (reflecting user fairness). To investigate how user preference similarity and active level skewness affect the optimal caching policy, we then provide a method to synthesize user preference for given content popularity and user active level. The analysis and simulation results show that exploiting user preference can improve both network performance and user fairness remarkably compared with priori works. The gain of exploiting user preference increases with user preference heterogeneity, user spatial locality, and skewness of user active level.
A Generative Model of 3D Object Layouts in Apartments

Understanding indoor scenes is an important task in computer vision. This task is typically ambiguous, so we require a strong prior, that captures the regularity of indoor environments. This is naturally expressed by a probabilistic model over 3D room layouts and geometry, reasoning over complex layouts in 3D space, including high-order spatial relations among objects. In this work, we construct such a model, trained on over 250000 human-designed rooms with 170 object classes. We conduct extensive experiments to show the quality of our model. First, we show that it generates samples that are plausible, by an extensive user study involving human comparisons of sampled layouts to ground-truth. Second, we demonstrate the value of incorporating spatial relationships between objects, by showing that this increases the plausibility of samples. Third, we show that our model generalises, rather than simply memorising its training set. Finally, we provide many examples of knowledge learnt by our model, such as support relationships, and common spatial relations between object classes.
A Neural-Symbolic Approach to Natural Language Tasks

Deep learning (DL) has in recent years been widely used in natural language processing (NLP) applications due to its superior performance. However, while natural languages are rich in grammatical structure, DL has not been able to explicitly represent and enforce such structures. This paper proposes a new architecture to bridge this gap by exploiting tensor product representations (TPR), a structured neural-symbolic framework developed in cognitive science over the past 20 years, with the aim of integrating DL with explicit language structures and rules. We call it the Tensor Product Generation Network (TPGN), and apply it to image captioning. The key ideas of TPGN are: 1) unsupervised learning of role-unbinding vectors of words via a TPR-based deep neural network, and 2) integration of TPR with typical DL architectures including Long Short-Term Memory (LSTM) models. The novelty of our approach lies in its ability to generate a sentence and extract partial grammatical structure of the sentence by using role-unbinding vectors, which are obtained in an unsupervised manner. Experimental results demonstrate the effectiveness of the proposed approach.
Evaluating Bayesian and L1 Approaches for Sparse Unsupervised Learning

The use of L1 regularisation for sparse learning has generated immense research interest, with many successful applications in diverse areas such as signal acquisition, image coding, genomics and collaborative filtering. While existing work highlights the many advantages of L1 methods, in this paper we find that L1 regularisation often dramatically under-performs in terms of predictive performance when compared to other methods for inferring sparsity. We focus on unsupervised latent variable models, and develop L1 minimising factor models, Bayesian variants of "L1", and Bayesian models with a stronger L0-like sparsity induced through spike-and-slab distributions. These spike-and-slab Bayesian factor models encourage sparsity while accounting for uncertainty in a principled manner, and avoid unnecessary shrinkage of non-zero values. We demonstrate on a number of data sets that in practice spike-and-slab Bayesian methods outperform L1 minimisation, even on a computational budget. We thus highlight the need to re-assess the wide use of L1 methods in sparsity-reliant applications, particularly when we care about generalising to previously unseen data, and provide an alternative that, over many varying conditions, provides improved generalisation performance.
A Probabilistic Framework for Multi-Label Learning with Unseen Labels

We present a probabilistic framework for multi-label learning for the setting when the test data may require predicting labels that were not available at training time (i.e., the zero-shot learning setting). We develop a probabilistic model that leverages the co-occurrence statistics of the labels via a joint generative model for the label matrix (which denotes the label presence/absence for each training example) and for the label co-occurrence matrix (which denotes how many times a pair of labels co-occurs with each other). In addition to handling the unseen labels at test time, leveraging the co-occurrence information may also help in the standard multi-label learning setting, especially if the number of training examples is very small and/or the label matrix of training examples has a large fraction of missing entries. Our experimental results demonstrate the efﬁcacy of our model in handling unseen labels.
Multi-Path Transmission, Selection and Handover Mechanism for High-Quality VoIP

With the development of audio encoding standards and IP technology, voice over IP (VoIP) is becoming quite popular. However, high-quality voice data over current networks without QoS Support, such as Internet and UMTS, still posses several challenging problems because of the adverse effects caused by network bandwidth restrictions and complex dynamics. One approach to provide QoS for VoIP applications over the wireless networks is to use multiple paths to deliver VoIP data destined for a particular receiver, i.e., this data is fragmented into packets and the different packets take alternate routes to the receiver. One advantage of this approach is that the complexity of QoS provision can be pushed to the network edge and hence improve the scalability and deployment characteristics while at the same time provide a certain level of QoS guarantees. The common view among researchers of the next generation mobile communication is that it will be a heterogeneous network environment, offering seamless services such as VoIP across multiple wireless access technologies. In the future there will be more multimode devices which can access multiple radio access networks. Moreover in the future we will see greater overlap between the coverage provided by the differing access technologies as Fig. 1. A host is multi-homed if it can be addressed by multiple IP addresses, as is the common case when the host has multiple network interfaces. Multi-homing is increasingly economically feasible and can be expected to be the rule rather than the exception in the near future. This chapter proposed a novel transport layer solution cmpSCTP that aims at exploiting SCTP’s multi-homing capability by selecting several paths among multiple available network interfaces to improve data transfer rate to the same multi-homed device. As such, it is naturally leads to another two new issues: (1) How to select most appropriate paths for CMT. As different paths are likely to overlap each other and even share bottleneck which lurks behind the IP/network layer topology, it is necessary to fall back on end-to-end probes to estimate this correlation by analyzing path characteristics so that we can select multiple independent paths as much as possible; (2) How to seamless handover paths for mobility. Using cmpSCTP’s flexible path management capability, we may switch the flow between multiple paths automatically to realize the seamless handover called Latent Handover, which is flow-oriented and switches the traffic to the new path progressively to make the handover process unconscious to users or upper layer applications, especially for real-time