Reviewerly/exHarmony huggingface dataset:
Graph routing between capsules

Routing methods in capsule networks often learn a hierarchical relationship for capsules in successive layers, but the intra-relation between capsules in the same layer is less studied, while this intra-relation is a key factor for the semantic understanding in text data. Therefore, in this paper, we introduce a new capsule network with graph routing to learn both relationships, where capsules in each layer are treated as the nodes of a graph. We investigate strategies to yield adjacency and degree matrix with three different distances from a layer of capsules, and propose the graph routing mechanism between those capsules. We validate our approach on five text classification datasets, and our findings suggest that the approach combining bottom-up routing and top-down attention performs the best. Such an approach demonstrates generalization capability across datasets. Compared to the state-of-the-art routing methods, the improvements in accuracy in the five datasets we used were 0.82, 0.39, 0.07, 1.01, and 0.02, respectively.
An Episodic Learning Network for Text Detection on Human Bodies in Sports Images

Due to the proliferation of sports-related multimedia content on the WWW, effective visual search and retrieval present interesting research challenges. These are caused by poor image quality, a wide range of possible camera points of view, pose variations on the part of athletes engaged in playing a sport, deformations of text appearing on sports person’s clothing and uniforms in motion, occlusions caused by other objects, etc. To address these challenges, this paper presents a new method for detecting text on human bodies in sports images. Unlike most existing methods, which attempt to exploit locations of a player’s torso, face, and skin, we propose an end-to-end episodic learning approach that employs inductive learning criteria for detecting clothing regions in an image, which are, in turn, then used for text detection. Our method integrates a Residual Network (ResNet) and Pyramidal Pooling Module (PPM) for generating a spatial attention map. The Progressive Scalable Expansion Algorithm (PSE) is adapted for text detection from these regions. Experimental results on our own dataset as well as several benchmarks (like RBNR and MMM which contain images of runners in marathons, and Re-ID which is a person re-identification dataset) demonstrate that the proposed method outperforms existing methods in terms of precision and F1-score. We also present results for sports images chosen from natural scene text detection datasets such as CTW1500 and MS-COCO to show the proposed method is effective and reliable across a range of inputs.
Efficient Graph Summarization using Weighted LSH at Billion-Scale

Summarizing graphs is of paramount importance due to diverse applications of large-scale graph analysis. A popular family of summarization methods is the group-based approach. The general idea consists of merging nodes of the original graph into supernodes of the summary graph, encoding original edges into superedges/correction set edges, and dropping certain superedges or correction set edges (for lossy summarization). The current state of the art has several steps in its computation that are serious bottlenecks in terms of running time and scalability. In this work, we propose algorithm LDME, a correction set based graph summarization algorithm that produces compact output representations in a fast and scalable manner. To achieve this, we introduce (1) weighted locality sensitive hashing to drastically reduce the number comparisons required to find good node merges, (2) an efficient way to compute the best quality merges that produces more compact outputs, and (3) a new sort-based encoding algorithm that is faster and more robust. More interestingly, our algorithm provides performance tuning settings to allow the option of trading compression for running time. On high compression settings, LDME achieves compression equal to or better than the state of the art with up to 53x speedup in running time. On high speed settings, LDME achieves up to two orders of magnitude speedup with only slightly lower compression.
Understanding Object Dynamics for Interactive Image-to-Video Synthesis

What would be the effect of locally poking a static scene? We present an approach that learns naturally-looking global articulations caused by a local manipulation at a pixel level. Training requires only videos of moving objects but no information of the underlying manipulation of the physical scene. Our generative model learns to infer natural object dynamics as a response to user interaction and learns about the interrelations between different object body regions. Given a static image of an object and a local poking of a pixel, the approach then predicts how the object would deform over time. In contrast to existing work on video prediction, we do not synthesize arbitrary realistic videos but enable local interactive control of the deformation. Our model is not restricted to particular object categories and can transfer dynamics onto novel unseen object instances. Extensive experiments on diverse objects demonstrate the effectiveness of our approach compared to common video prediction frameworks. Project page is available at https://bit.ly/3cxfA2L.
Striking the Right Balance: Recall Loss for Semantic Segmentation

Class imbalance is a fundamental problem in computer vision applications such as semantic segmentation. Specifically, uneven class distributions in a training dataset often result in unsatisfactory performance on under-represented classes. Many works have proposed to weight the standard cross entropy loss function with pre-computed weights based on class statistics, such as the number of samples and class margins. There are two major drawbacks to these methods: 1) constantly up-weighting minority classes can introduce excessive false positives in semantic segmentation; 2) a minority class is not necessarily a hard class. The consequence is low precision due to excessive false positives. In this regard, we propose a hard-class mining loss by reshaping the vanilla cross entropy loss such that it weights the loss for each class dynamically based on instantaneous recall performance. We show that the novel recall loss changes gradually between the standard cross entropy loss and the inverse frequency weighted loss. Recall loss also leads to improved mean accuracy while offering competitive mean Intersection over Union (IoU) performance. On Synthia dataset <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Synthia-Sequence Summer split, recall loss achieves 9% relative improvement on mean accuracy with competitive mean IoU using DeepLab-ResNet18 compared to the cross entropy loss. Code available at https://github.com/PotatoTian/recall-semseg.
Multi-task Pruning via Filter Index Sharing: A Many-Objective Optimization Approach

Abstract State-of-the-art deep neural network plays an increasingly important role in artificial intelligence, while the huge number of parameters in networks brings high memory cost and computational complexity. To solve this problem, filter pruning is widely used for neural network compression and acceleration. However, existing algorithms focus mainly on pruning single model, and few results are available to multi-task pruning that is capable of pruning multi-model and promoting the learning performance. By utilizing the filter sharing technique, this paper aimed to establish a multi-task pruning framework for simultaneously pruning and merging filters in multi-task networks. An optimization problem of selecting the important filters is solved by developing a many-objective optimization algorithm where three criteria are adopted as objectives for the many-objective optimization problem. With the purpose of keeping the network structure, an index matrix is introduced to regulate the information sharing during multi-task training. The proposed multi-task pruning algorithm is quite flexible that can be performed with either adaptive or pre-specified pruning rates. Extensive experiments are performed to verify the applicability and superiority of the proposed method on both single-task and multi-task pruning.
A Region-and-Trajectory Movement Matching for Multiple Turn-counts at Road Intersection on Edge Device

In intelligent traffic systems, vehicle detection and counting have become an important task. The counting information is essential for reducing traffic congestion and improving traffic signal capability. Traditional methods have been focusing on counting vehicles in a single frame or consecutive frames. However, they have not yet considered the movement of interest (MOI) of the vehicles moving in different lanes and directions. This paper proposes a region-and-trajectory movement matching method that aims to detect and count vehicles for each movement on the road. First, the YOLOv5 detection model is used to detect candidate vehicles in the region of interest (ROI). Second, the SORT tracking method associates vehicles of the same instance in consecutive images to create tracked trajectories. Then, the counting method using the combination of MOI regions and predefined movement tracks. Each tracked trajectory is assigned to the corresponding movement id and is outputted to the result file. The efficiency and effectiveness of the proposed method have been evaluated and ranked 3rd on AI City Challenge 2021 Track 1 leaderboard. Further experiments showed that the method could achieve around 120 fps on an NVIDIA Quadro RTX 8000 and 20 fps on an NVIDIA Jetson Xavier AGX.
Shape from apparent contours for bent pipes with constant diameter under perspective projection

• The single-view perspective projection model of a pipe with constant diameter is established. • Projected axis extraction method based on two facing apparent contours is developed. • A trinocular vision system is adopted to realize the 3D reconstruction of pipe axis. • After the pipe axis is reconstructed, the pipe radius can also be measured. In perspective projection images, apparent contours are important bases for recovering the three-dimensional (3D) shapes of constant diameter pipes (CDP). In order to complete the 3D reconstruction of CDP under perspective projection, this article first analyzes the single-view perspective projection model of CDP in detail. On this basis, coupled point pairs searching and projected axis extraction methods based on two facing apparent contours are proposed. After that, stereo vision technology is adopted to realize the 3D reconstruction of axis and the measurement of radius. Finally, the experiments with both synthetic and real data show that the proposed method has higher CDP reconstruction accuracy compared with the conventional method processed under orthogonal projection.
Enabling Long-Term Cooperation in Cross-Silo Federated Learning: A Repeated Game Perspective

Cross-silo federated learning (FL) is a distributed learning approach where clients of the same interest train a global model cooperatively while keeping their local data private. The success of a cross-silo FL process requires active participation of many clients. Different from cross-device FL, clients in cross-silo FL are usually organizations or companies which may execute multiple cross-silo FL processes repeatedly due to their time-varying local data sets, and aim to optimize their long-term benefits by selfishly choosing their participation levels. While there has been some work on incentivizing clients to join FL, the analysis of clients’ long-term selfish participation behaviors in cross-silo FL remains largely unexplored. In this paper, we analyze the selfish participation behaviors of heterogeneous clients in cross-silo FL. Specifically, we model clients’ long-term selfish participation behaviors as an infinitely repeated game, with the stage game being a selfish participation game in one cross-silo FL process (SPFL). For the stage game SPFL, we derive the unique Nash equilibrium (NE), and propose a distributed algorithm for each client to calculate its equilibrium participation strategy. We show that at the NE, clients fall into at most three categories: (i) <i>free riders</i> who do not perform local model training, (ii) a unique <i>partial contributor</i> (if exists) who performs model training with part of its local data, and (iii) <i>contributors</i> who perform model training with all their local data. The existence of free riders has a detrimental effect on achieving a good global model and sustaining other clients’ long-term participation. For the long-term interactions among clients, we derive a cooperative strategy for clients which minimizes the number of free riders while increasing the amount of local data for model training. We show that enforced by a punishment strategy, such a cooperative strategy is a subgame perfect Nash equilibrium (SPNE) of the infinitely repeated game, under which some clients who are free riders at the NE of the stage game choose to be (partial) contributors. We further propose an algorithm to calculate the optimal SPNE which minimizes the number of free riders while maximizing the amount of local data for model training. Simulation results show that our derived optimal SPNE can effectively reduce the number of free riders by up to <inline-formula><tex-math notation="LaTeX">$99.3\%$</tex-math></inline-formula> and increase the amount of local data for model training by up to <inline-formula><tex-math notation="LaTeX">$82.3\%$</tex-math></inline-formula> .
Variational AutoEncoder for Reference based Image Super-Resolution

In this paper, we propose a novel reference based image super-resolution approach via Variational AutoEncoder (RefVAE). Existing state-of-the-art methods mainly focus on single image super-resolution which cannot perform well on large upsampling factors, e.g., 8×. We propose a reference based image super-resolution, for which any arbitrary image can act as a reference for super-resolution. Even using random map or low-resolution image itself, the proposed RefVAE can transfer the knowledge from the reference to the super-resolved images. Depending upon different references, the proposed method can generate different versions of super-resolved images from a hidden super- resolution space. Besides using different datasets for some standard evaluations with PSNR and SSIM, we also took part in the NTIRE2021 SR Space challenge [29] and have provided results of the randomness evaluation of our approach. Compared to other state-of-the-art methods, our approach achieves higher diverse scores.
Learning Deep Classifiers Consistent with Fine-Grained Novelty Detection

The problem of novelty detection in fine-grained visual classification (FGVC) is considered. An integrated understanding of the probabilistic and distance-based approaches to novelty detection is developed within the frame-work of convolutional neural networks (CNNs). It is shown that softmax CNN classifiers are inconsistent with novelty detection, because their learned class-conditional distributions and associated distance metrics are unidentifiable. A new regularization constraint, the class-conditional Gaussianity loss, is then proposed to eliminate this unidentifiability, and enforce Gaussian class-conditional distributions. This enables training Novelty Detection Consistent Classifiers (NDCCs) that are jointly optimal for classification and novelty detection. Empirical evaluations show that NDCCs achieve significant improvements over the state-of-the-art on both small- and large-scale FGVC datasets.
Compact and Effective Representations for Sketch-based Image Retrieval

Sketch-based image retrieval (SBIR) has undergone an increasing interest in the community of computer vision bringing high impact in real applications. For instance, SBIR brings an increased benefit to eCommerce search engines because it allows users to formulate a query just by drawing what they need to buy. However, current methods showing high precision in retrieval work in a high dimensional space, which negatively affects aspects like memory consumption and time processing. Although some authors have also proposed compact representations, these drastically degrade the performance in a low dimension. Therefore in this work, we present different results of evaluating methods for producing compact embeddings in the context of sketch-based image retrieval. Our main interest is in strategies aiming to keep the local structure of the original space. The recent unsupervised local-topology preserving dimension reduction method UMAP fits our requirements and shows outstanding performance, improving even the precision achieved by SOTA methods. We evaluate six methods in two different datasets. We use Flickr15K and eCommerce datasets; the latter is another contribution of this work. We show that UMAP allows us to have feature vectors of 16 bytes improving precision by more than 35%.
Blur, Noise, and Compression Robust Generative Adversarial Networks

Generative adversarial networks (GANs) have gained considerable attention owing to their ability to reproduce images. However, they can recreate training images faithfully despite image degradation in the form of blur, noise, and compression, generating similarly degraded images. To solve this problem, the recently proposed noise robust GAN (NR-GAN) provides a partial solution by demonstrating the ability to learn a clean image generator directly from noisy images using a two-generator model comprising image and noise generators. However, its application is limited to noise, which is relatively easy to decompose owing to its additive and reversible characteristics, and its application to irreversible image degradation, in the form of blur, compression, and combination of all, remains a challenge. To address these problems, we propose blur, noise, and compression robust GAN (BNCR-GAN) that can learn a clean image generator directly from degraded images without knowledge of degradation parameters (e.g., blur kernel types, noise amounts, or quality factor values). Inspired by NR-GAN, BNCR-GAN uses a multiple-generator model composed of image, blur-kernel, noise, and quality-factor generators. However, in contrast to NR-GAN, to address irreversible characteristics, we introduce masking architectures adjusting degradation strength values in a data-driven manner using bypasses before and after degradation. Furthermore, to suppress uncertainty caused by the combination of blur, noise, and compression, we introduce adaptive consistency losses imposing consistency between irreversible degradation processes according to the degradation strengths. We demonstrate the effectiveness of BNCR-GAN through large-scale comparative studies on CIFAR-10 and a generality analysis on FFHQ. In addition, we demonstrate the applicability of BNCR-GAN in image restoration.
Exploring the Real-Time Touchless Hand Interaction and Intelligent Agents in Augmented Reality Learning Applications

During the last decade, there has been a surge in research studies exploring the adoption of Augmented Reality (AR) in educational settings. Within these multiple research studies, AR's capability to extend the teaching and learning environment with augmented 3D learning objects with enhanced interactive capabilities have been demonstrated. This new technology has not been widely adopted in the mainstream but with the recent unprecedented circumstances of COVID-19, there has been an increasing societal willingness to adopt these technologies. AR has been a desirable technology due to its inherent touchless nature which facilitates social distancing at this time but AR applications crucially offer so much more. They can provide interactive functionality through augmentation of the teaching and learning environment within an immersive user experience including 3D interactions with learning objects, gestures, hand interaction, tangible and multi-modal interaction. This paper presents the results of a review of touchless interaction studies in educational applications and proposes the implementation of real-time touchless hand interaction within kinesthetic learning and utilization of machine learning agents. The architecture of two AR applications with real-time hand interaction and machine learning agents are demonstrated within this paper enabling engaged kinesthetic learning as an alternative learning interface.
The different types of reports produced in digital forensic investigations

The importance of ensuring the results of any digital forensic (DF) examination are effectively communicated cannot be understated. In most cases, this communication will be done via written report, yet despite this there is arguably limited best practice guidance available which is specific for this field in regards to report construction. Poor reporting practices in DF are likely to undermine the reliability of evidence provided across this field, where there is a need for formalised guidance regarding the requirements for effective DF report construction; this should not be a task left solely to each individual practitioner to determine without instruction. For this, the field of DF should look to the wider forensic community and the existing work in this area for support. In line with many other 'traditional' forensic science types, a DF practitioner can be commissioned to report in one of three ways - 'technical', 'investigative' or 'evaluative', where each reporting type maintains a specific purpose and interpretative-context, determined by the examination workflow undertaken by a practitioner following client instruction. This work draws upon guidance set out in fundamental forensic science reporting literature in order to describe each reporting type in turn, outlining their scope, content and construction requirements in an attempt to provide support for the DF field.
Authenticity perceptions in virtual environments

Purpose This paper aims to examine three distinct aspects of authenticity that pre-service teachers (PSTs) experience when they engage with virtual classroom environments to develop their content-intensive instructional practice – task authenticity, student avatar authenticity and performance authenticity – and their perceptions about the usefulness of the simulated teaching experience to support their learning. Design/methodology/approach This paper explored these conceptions of authenticity and usefulness within a larger research study whose goal was to develop virtual environment tools to help elementary PSTs learn how to engage in one ambitious teaching practice: facilitating discussions that engage students in argumentation. To examine these aspects of authenticity and usefulness, this paper used a general qualitative deductive analysis approach to examine data from 104 interviews with 26 case study teachers and examined patterns in PSTs’ perceptions within and across interviews and authenticity aspects. Findings While these PSTs strongly value the utility of these tools to support their learning, findings point to variation in their perceptions of authenticity. Findings showed that most PSTs perceived the tasks as an authentic representation of the work of teaching. However, their perceptions of task authenticity did not always align with their perceptions of avatar or performance authenticity. Originality/value This paper argues that these three aspects of authenticity relate to, but expand upon, the broader notions of presence and plausibility noted in the literature on virtual environments and should be taken up more directly in future studies of users’ perceptions of virtual environments both within and outside of educational contexts.