Reviewerly/exHarmony huggingface dataset:
Progressively Complementary Network for Fisheye Image Rectification Using Appearance Flow

Distortion rectification is often required for fisheye images. The generation-based method is one mainstream solution due to its label-free property, but its naive skip-connection and overburdened decoder will cause blur and incomplete correction. First, the skip-connection directly transfers the image features, which may introduce distortion and cause incomplete correction. Second, the decoder is overburdened during simultaneously reconstructing the content and structure of the image, resulting in vague performance. To solve these two problems, in this paper, we focus on the interpretable correction mechanism of the distortion rectification network and propose a feature-level correction scheme. We embed a correction layer in skip-connection and leverage the appearance flows in different layers to pre-correct the image features. Consequently, the decoder can easily reconstruct a plausible result with the remaining distortion-less information. In addition, we propose a parallel complementary structure. It effectively reduces the burden of the decoder by separating content reconstruction and structure correction. Subjective and objective experiment results on different datasets demonstrate the superiority of our method.
Suspicious Activity Recognition Using Proposed Deep L4-Branched-Actionnet With Entropy Coded Ant Colony System Optimization

Intelligent visual surveillance systems are attracting much attention from research and industry. The invention of smart surveillance cameras with greater processing power has now been the leading stakeholder, making it conceivable to design intelligent visual surveillance systems. It is possible to assure the safety of people in both homes and public places. This work aims to distinguish the suspicious activities for surveillance environments. For this, a 63 layers deep CNN model is suggested and named “L4-Branched-ActionNet”. The suggested CNN structure is centered on the alteration of AlexNet with added four blanched sub-structures. The developed framework is first transformed into a pre-trained framework by conducting its training on an object detection dataset called CIFAR-100 with the SoftMax function. The dataset for suspicious activity recognition is then forwarded to this pretrained model for feature acquisition. The acquired deep features are subjected to feature subset optimization. These extracted features are first coded by applying entropy and then an ant colony system (ACS) is utilized on the entropy-based coded features for optimization. The configured features are then fed into numerous SVM and KNN based classification models. The cubic SVM has the highest efficiency scores, with a performance of 0.9924 in order of accuracy. The proposed model is also validated on the Weizmann action dataset and attained an accuracy of 0.9796. The successful findings indicate the suggested work’s soundness.
Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association

Nowadays, we have witnessed the early progress on learning the association between voice and face automatically, which brings a new wave of studies to the computer vision community. However, most of the prior arts along this line (a) merely adopt local information to perform modality alignment and (b) ignore the diversity of learning difficulty across different subjects. In this paper, we propose a novel framework to jointly address the above-mentioned issues. Targeting at (a), we propose a two-level modality alignment loss where both global and local information are considered. Compared with the existing methods, we introduce a global loss into the modality alignment process. The global component of the loss is driven by the identity classification. Theoretically, we show that minimizing the loss could maximize the distance between embeddings across different identities while minimizing the distance between embeddings belonging to the same identity, in a global sense (instead of a mini-batch). Targeting at (b), we propose a dynamic reweighting scheme to better explore the hard but valuable identities while filtering out the unlearnable identities. Experiments show that the proposed method outperforms the previous methods in multiple settings, including voice-face matching, verification and retrieval.
Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching

Image-text matching is an important multi-modal task with massive applications. It tries to match the image and the text with similar semantic information. Existing approaches do not explicitly transform the different modalities into a common space. Meanwhile, the attention mechanism which is widely used in image-text matching models does not have supervision. We propose a novel attention scheme which projects the image and text embedding into a common space and optimises the attention weights directly towards the evaluation metrics. The proposed attention scheme can be considered as a kind of supervised attention and requiring no additional annotations. It is trained via a novel Discrete-continuous action space policy gradient algorithm, which is more effective in modelling complex action space than previous continuous action space policy gradient. We evaluate the proposed methods on two widely-used benchmark datasets: Flickr30k and MS-COCO, outperforming the previous approaches by a large margin.
Rank-One Prior: Toward Real-Time Scene Recovery

Scene recovery is a fundamental imaging task for several practical applications, e.g., video surveillance and autonomous vehicles, etc. To improve visual quality under different weather/imaging conditions, we propose a real-time light correction method to recover the degraded scenes in the cases of sandstorms, underwater, and haze. The heart of our work is that we propose an intensity projection strategy to estimate the transmission. This strategy is motivated by a straightforward rank-one transmission prior. The complexity of transmission estimation is O(N ) where N is the size of the single image. Then we can recover the scene in real-time. Comprehensive experiments on different types of weather/imaging conditions illustrate that our method outperforms competitively several state-of-the-art imaging methods in terms of efficiency and robustness.
MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning

Distributed synchronous stochastic gradient descent has been widely used to train deep neural networks (DNNs) on computer clusters. With the increase of computational power, network communications generally limit the system scalability. Wait-free backpropagation (WFBP) is a popular solution to overlap communications with computations during the training process. In this article, we observe that many DNNs have a large number of layers with only a small amount of data to be communicated at each layer in distributed training, which could make WFBP inefficient. Based on the fact that merging some short communication tasks into a single one can reduce the overall communication time, we formulate an optimization problem to minimize the training time in pipelining communications and computations. We derive an optimal solution that can be solved efficiently without affecting the training performance. We then apply the solution to propose a distributed training algorithm named merged-gradient WFBP (MG-WFBP) and implement it in two platforms Caffe and PyTorch. Extensive experiments in three GPU clusters are conducted to verify the effectiveness of MG-WFBP. We further exploit trace-based simulations of 4 to 2048 GPUs to explore the potential scaling efficiency of MG-WFBP. Experimental results show that MG-WFBP achieves much better scaling performance than existing methods.
Efficient and decision boundary aware instance selection for support vector machines

Support vector machines (SVMs) are powerful classifiers that have high computational complexity in the training phase, which can limit their applicability to large datasets. An effective approach to address this limitation is to select a small subset of the most representative training samples such that desirable results can be obtained. In this study, a novel instance selection method called border point extraction based on locality-sensitive hashing (BPLSH) is designed. BPLSH preserves instances that are near the decision boundaries and eliminates nonessential ones. The performance of BPLSH is benchmarked against four approaches on different classification problems. The experimental results indicate that BPLSH outperforms the other methods in terms of classification accuracy, preservation rate, and execution time. The source code of BPLSH can be found in https://github.com/mohaslani/BPLSH.
An intelligent system for focused crawling from Big Data sources

Nowadays, the proper management of data is a key business enabler and booster for companies, so as to increase their competitiveness. Typically, companies hold massive amounts of data within their servers, which might include previously offered services, proposals, bids, and so on. They rely on their expert managers to manually analyse them in order to make strategic decisions. However, given the huge amount of information to be analysed and the necessity of making timely decisions, they often exploit a small amount of the available data, which often does not yield effective choices. For instance, this happens in the context of the e-procurement domain, where bids for new calls for tender are often formulated by looking at some past proposals from a company. Driven by an extensive experience on the e-procurement domain, in this paper we propose an intelligent system to support organisations in the focused crawling of artefacts (calls for tender, BIMs, equipment, policies, market trends, and so on) of interest from the web, semantically matching them against internal Big Data and knowledge sources, so as to let companies analysts make better strategic decisions. The novel contribution consists of a proper extension of the K-means algorithm used by a web crawler within the proposed system, and a semantic module exploiting search patterns to find relevant data within the crawled artefacts. The proposed solution has been implemented and extensively assessed in the e-procurement domain. It has been successively extended to other domains, such as robot programming, cloud providing, and several other domains. Since to the best of our knowledge in the literature do not exists similar systems, in order to prove its effectiveness we have compared its crawling component against similar crawlers, by plugging them within our system.
Performance Evaluation of an Earthquake Early Warning System in the 2019–2020 M6.0 Changning, Sichuan, China, Seismic Sequence

China is currently building a nationwide earthquake early warning system (EEWS) which will be completed in June 2023. Several regions have been selected as pilot areas for instrumentation, software system and dissemination verification. For these regions, their construction tasks will be completed in advance with trial runs being carried out in June 2021. Before the trial operation, we need to understand the actual processing capabilities of different EEWSs. In this work, we focus on the system deployed in Sichuan province and evaluate its real-time performance during the 2019–2020 M 6.0 Changning seismic sequence. This period was divided into two stages. The first stage was the time from the occurrence of the M 6.0 ( M w 5.7) mainshock (June 17, 2019) to the end of October 2019 with no MEMS-based stations around the Changning seismic sequence deployed and most of the broadband and short period seismic stations not upgraded to low latency streaming, and the second one was from the beginning of November 2019 to March 2021 with deployments of more than 700 MEMS-based stations and low latency upgrades of ∼30 seismic stations. Median errors for the epicentral locations, depths and magnitude estimations were almost the same for both stages, 1.5 ± 6.0 km, 0.0 ± 3.6 km and −0.1 ± 0.46 for the first stage and 2.3 ± 3.0 km, −3.0 ± 3.6 km and −0.2 ± 0.32 for the second one. However, an obvious underestimation of the magnitude for earthquakes with M 5.0 + occurring in the first stage was observed, which would be caused by the clipped waveforms, sensors deployed in short period seismic stations and MEMS-based stations, the adopted magnitude estimation method, and the method used to computer the network magnitude. The median reporting time was significantly improved from 10.5 ± 3.0 s after origin time for the first stage to 6.3 ± 3.5 s for the second stage because of introduction of newly deployed MEMS-based stations. The results obtained from the second stage indicate that the system has entered a stable operating stage and we can officially launch the trial operation in the pilot areas for public early warning services.
FedCT: Federated Collaborative Transfer for Recommendation

When a user starts exploring items from a new area of an e-commerce system, cross-domain recommendation techniques come into help by transferring the abundant knowledge from the user's familiar domains to this new domain. However, this solution usually requires direct information sharing between service providers on the cloud which may not always be available and brings privacy concerns. In this paper, we show that one can overcome these concerns through learning on edge devices such as smartphones and laptops. The cross-domain recommendation problem is formalized under a decentralized computing environment with multiple domain servers. And we identify two key challenges for this setting: the unavailability of direct transfer and the heterogeneity of the domain-specific user representations. We then propose to learn and maintain a decentralized user encoding on each user's personal space. The optimization follows a variational inference framework that maximizes the mutual information between the user's encoding and the domain-specific user information from all her interacted domains. Empirical studies on real-world datasets exhibit the effectiveness of our proposed framework on recommendation tasks and its superiority over domain-pairwise transfer models. The resulting system offers reduced communication cost and an efficient inference mechanism that does not depend on the number of involved domains, and it allows flexible plugin of domain-specific transfer models without significant interference on other domains.
VaB-AL: Incorporating Class Imbalance and Difficulty with Variational Bayes for Active Learning

Active Learning for discriminative models has largely been studied with the focus on individual samples, with less emphasis on how classes are distributed or which classes are hard to deal with. In this work, we show that this is harmful. We propose a method based on the Bayes’ rule, that can naturally incorporate class imbalance into the Active Learning framework. We derive that three terms should be considered together when estimating the probability of a classifier making a mistake for a given sample; i) probability of mislabelling a class, ii) likelihood of the data given a predicted class, and iii) the prior probability on the abundance of a predicted class. Implementing these terms requires a generative model and an intractable likelihood estimation. Therefore, we train a Variational Auto Encoder (VAE) for this purpose. To further tie the VAE with the classifier and facilitate VAE training, we use the classifiers’ deep feature representations as input to the VAE. By considering all three probabilities, among them, especially the data imbalance, we can substantially improve the potential of existing methods under limited data budget. We show that our method can be applied to classification tasks on multiple different datasets – including one that is a real-world dataset with heavy data imbalance – significantly outperforming the state of the art.
Software Architecture for ML-based Systems: What Exists and What Lies Ahead

The increasing usage of machine learning (ML) coupled with the software architectural challenges of the modern era has resulted in two broad research areas: i) software architecture for ML-based systems, which focuses on developing architectural techniques for better developing ML-based software systems, and ii) ML for software architectures, which focuses on developing ML techniques to better architect traditional software systems. In this work, we focus on the former side of the spectrum with a goal to highlight the different architecting practices that exist in the current scenario for architecting ML-based software systems. We identify four key areas of software architecture that need the attention of both the ML and software practitioners to better define a standard set of practices for architecting ML-based software systems. We base these areas in light of our experience in architecting an ML-based software system for solving queuing challenges in one of the largest museums in Italy.
Immersive virtual reality (VR) for digital media making: transmediation is key

The rapid evolution of virtual reality (VR) technologies and their adoption for learning opens up new possibilities for shifting semiotic content across modes, with underexplored scope for transmediating content in visual, haptic, and auditory ways in immersive media literacy practices. This research investigated users’ creative digital designing involving a popular, three-dimensional virtual painting program with upper elementary students who used a VR head mounted display and sensors. The analysis attended to how students transmediated the same story across written, verbal, and virtual painting modes, tracing key themes of the students’ virtual experience: (i) immersion and three-dimensionality, (ii) subjective presence, (iii) sensory illusion, and (iv) interactivity with motion tracking. Students reassembled and shifted narrative content, sometimes seamlessly, while experiencing ambiguity and complexity about three-dimensional representation in an immersive world. Producing stories across modes invited adaptation and generative thinking to overcome the lack of equivalence between writing, drawing, and virtual painting modes.
Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation

We propose Joint-DetNAS, a unified NAS framework for object detection, which integrates 3 key components: Neural Architecture Search, pruning, and Knowledge Distillation. Instead of naively pipelining these techniques, our Joint-DetNAS optimizes them jointly. The algorithm consists of two core processes: student morphism optimizes the student’s architecture and removes the redundant parameters, while dynamic distillation aims to find the optimal matching teacher. For student morphism, weight inheritance strategy is adopted, allowing the student to flexibly update its architecture while fully utilize the predecessor’s weights, which considerably accelerates the search; To facilitate dynamic distillation, an elastic teacher pool is trained via integrated progressive shrinking strategy, from which teacher detectors can be sampled without additional cost in subsequent searches. Given a base detector as the input, our algorithm directly outputs the derived student detector with high performance without additional training. Experiments demonstrate that our Joint-DetNAS outperforms the naive pipelining approach by a great margin. Given a classic R101-FPN as the base detector, Joint-DetNAS is able to boost its mAP from 41.4 to 43.9 on MS COCO and reduce the latency by 47%, which is on par with the SOTA EfficientDet while requiring less search cost. We hope our proposed method can provide the community with a new way of jointly optimizing NAS, KD and pruning.
DotSCN: Group Re-Identification via Domain-Transferred Single and Couple Representation Learning

Group re-identification (G-ReID) is an important yet less-studied task. Its challenges not only lie in appearance changes of individuals, but also involve group layout and membership changes. To address these issues, the key task of G-ReID is to learn group representations robust to such changes. Nevertheless, unlike ReID tasks, there still lacks comprehensive publicly available G-ReID datasets, making it difficult to learn effective representations using deep learning models. In this article, we propose a Domain-Transferred Single and Couple Representation Learning Network (DotSCN). Its merits are two aspects: 1) Owing to the lack of labelled training samples for G-ReID, existing G-ReID methods mainly rely on unsatisfactory hand-crafted features. To gain the power of deep learning models in representation learning, we first treat a group as a collection of multiple individuals and propose transferring the representation of individuals learned from an existing labeled ReID dataset to a target G-ReID domain without a suitable training dataset. 2) Taking into account the neighborhood relationship in a group, we further propose learning a novel couple representation between two group members, that achieves better discriminative power in G-ReID tasks. In addition, we propose a weight learning method to adaptively fuse the domain-transferred individual and couple representations based on an L-shape prior. Extensive experimental results demonstrate the effectiveness of our approach that significantly outperforms state-of-the-art methods by 11.7% CMC-1 on the Road Group dataset and by 39.0% CMC-1 on the DukeMCMT dataset.
Guided Integrated Gradients: an Adaptive Path Method for Removing Noise

Integrated Gradients (IG) [29] is a commonly used feature attribution method for deep neural networks. While IG has many desirable properties, the method often produces spurious/noisy pixel attributions in regions that are not related to the predicted class when applied to visual models. While this has been previously noted [27], most existing solutions [25], [17] are aimed at addressing the symptoms by explicitly reducing the noise in the resulting attributions. In this work, we show that one of the causes of the problem is the accumulation of noise along the IG path. To minimize the effect of this source of noise, we propose adapting the attribution path itself - conditioning the path not just on the image but also on the model being explained. We introduce Adaptive Path Methods (APMs) as a generalization of path methods, and Guided IG as a specific instance of an APM. Empirically, Guided IG creates saliency maps better aligned with the model's prediction and the input image that is being explained. We show through qualitative and quantitative experiments that Guided IG outperforms other, related methods in nearly every experiment.
Photonic Integrated Reconfigurable Linear Processors as Neural Network Accelerators

Reconfigurable linear optical processors can be used to perform linear transformations and are instrumental in effectively computing matrix–vector multiplications required in each neural network layer. In this paper, we characterize and compare two thermally tuned photonic integrated processors realized in silicon-on-insulator and silicon nitride platforms suited for extracting feature maps in convolutional neural networks. The reduction in bit resolution when crossing the processor is mainly due to optical losses, in the range 2.3–3.3 for the silicon-on-insulator chip and in the range 1.3–2.4 for the silicon nitride chip. However, the lower extinction ratio of Mach–Zehnder elements in the latter platform limits their expressivity (i.e., the capacity to implement any transformation) to 75%, compared to 97% of the former. Finally, the silicon-on-insulator processor outperforms the silicon nitride one in terms of footprint and energy efficiency.