Reviewerly/exHarmony huggingface dataset:
Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation

This article addresses the problem of distilling knowledge from a large teacher model to a slim student network for LiDAR semantic segmentation. Directly employing previous distillation approaches yields inferior results due to the intrinsic challenges of point cloud, i.e., sparsity, randomness and varying density. To tackle the aforementioned problems, we propose the Point-to-Voxel Knowledge Distillation (PVD), which transfers the hidden knowledge from both point level and voxel level. Specifically, we first leverage both the pointwise and voxelwise output distillation to complement the sparse supervision signals. Then, to better exploit the structural information, we divide the whole point cloud into several supervoxels and design a difficulty-aware sampling strategy to more frequently sample supervoxels containing less-frequent classes and faraway objects. On these supervoxels, we propose inter-point and inter-voxel affinity distillation, where the similarity information between points and voxels can help the student model better capture the structural information of the surrounding environment. We conduct extensive experiments on two popular LiDAR segmentation benchmarks, i.e., nuScenes [3] and SemanticKITTI [1]. On both benchmarks, our PVD-consistently outperforms previous distillation approaches by a large margin on three representative backbones, i.e., Cylinder3D [36], [37], SPVNAS [25] and MinkowskiNet [5]. Notably, on the challenging nuScenes and SemanticKITTI datasets, our method can achieve roughly 75% MACs reduction and 2× speedup on the competitive Cylinder3D model and rank 1st on the SemanticKITTI leaderboard among all published algorithms <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://competitions.codalab.org/competitions/20331#results (single-scan competition) till 2021-11-18 04:00 Pacific Time, and our method is termed Point-Voxel-KD. Our method (PV-KD) ranks 3rd on the multi-scan challenge till 2021-12-1 00:00 Pacific Time.. Our code is available at https://github.com/cardwing/Codes-for-PVKD.
Unified Transformer Tracker for Object Tracking

As an important area in computer vision, object tracking has formed two separate communities that respectively study Single Object Tracking (SOT) and Multiple Object Tracking (MOT). However, current methods in one tracking scenario are not easily adapted to the other due to the divergent training datasets and tracking objects of both tasks. Although UniTrack [45] demonstrates that a shared appearance model with multiple heads can be used to tackle individual tracking tasks, it fails to exploit the large-scale tracking datasets for training and performs poorly on the single object tracking. In this work, we present the Unified Transformer Tracker (UTT) to address tracking problems in different scenarios with one paradigm. A track transformer is developed in our UTT to track the target in both SOT and MOT where the correlation between the target feature and the tracking frame feature is exploited to localize the target. We demonstrate that both SOT and MOT tasks can be solved within this framework, and the model can be simultaneously end-to-end trained by alternatively optimizing the SOT and MOT objectives on the datasets of individual tasks. Extensive experiments are conducted on several benchmarks with a unified model trained on both SOT and MOT datasets.
FlowFormer: A Transformer Architecture for Optical Flow

We introduce optical Flow transFormer, dubbed as FlowFormer, a transformer-based neural network architecture for learning optical flow. FlowFormer tokenizes the 4D cost volume built from an image pair, encodes the cost tokens into a cost memory with alternate-group transformer (AGT) layers in a novel latent space, and decodes the cost memory via a recurrent transformer decoder with dynamic positional cost queries. On the Sintel benchmark, FlowFormer achieves 1.144 and 2.183 average end-ponit-error (AEPE) on the clean and final pass, a 17.6% and 11.6% error reduction from the best published result (1.388 and 2.47). Besides, FlowFormer also achieves strong generalization performance. Without being trained on Sintel, FlowFormer achieves 0.95 AEPE on the Sintel training set clean pass, outperforming the best published result (1.29) by 26.9%.
Scene Text Recognition with Permuted Autoregressive Sequence Models

Context-aware STR methods typically use internal autoregressive (AR) language models (LM). Inherent limitations of AR models motivated two-stage methods which employ an external LM. The conditional independence of the external LM on the input image may cause it to erroneously rectify correct predictions, leading to significant inefficiencies. Our method, PARSeq, learns an ensemble of internal AR LMs with shared weights using Permutation Language Modeling. It unifies context-free non-AR and context-aware AR inference, and iterative refinement using bidirectional context. Using synthetic training data, PARSeq achieves state-of-the-art (SOTA) results in STR benchmarks (91.9% accuracy) and more challenging datasets. It establishes new SOTA results (96.0% accuracy) when trained on real data. PARSeq is optimal on accuracy vs parameter count, FLOPS, and latency because of its simple, unified structure and parallel token processing. Due to its extensive use of attention, it is robust on arbitrarily-oriented text, which is common in real-world images. Code, pretrained weights, and data are available at: https://github.com/baudm/parseq .
Lite Vision Transformer with Enhanced Self-Attention

Despite the impressive representation capacity of vision transformer models, current light-weight vision transformer models still suffer from inconsistent and incorrect dense predictions at local regions. We suspect that the power of their self-attention mechanism is limited in shallower and thinner networks. We propose Lite Vision Transformer (LVT), a novel light-weight transformer network with two enhanced self-attention mechanisms to improve the model performances for mobile deployment. For the low-level features, we introduce Convolutional Self-Attention (CSA). Unlike previous approaches of merging convolution and self-attention, CSA introduces local self-attention into the convolution within a kernel of size <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$3\times 3$</tex> to enrich low-level features in the first stage of LVT. For the high-level features, we propose Recursive Atrous Self-Attention (RASA), which utilizes the multi-scale context when calculating the similarity map and a recursive mechanism to increase the representation capability with marginal extra parameter cost. The superiority of LVT is demonstrated on ImageNet recognition, ADE20K semantic segmentation, and COCO panoptic segmentation. The code is made publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://github.com/Chenglin-Yang/LVT.
The illusion of the metaverse and meta-economy

This paper provides (i) a review of the existing literature on the metaverse and (ii) an empirical assessment of the current state of the Web3 meta-economy, with the focus on economic governance and metaverse commerce. We have analysed the entire Web3 metaverse niche, i.e. both the 196 available metaverse fungible tokens and all the non-fungible token (NFT) transactions belonging to the metaverse marketplace. Our results showed that economic governance is based on metaverse tokens that cannot be defined as reliable virtual currencies due to their explosive behaviour, negative performance, and higher volatility compared to traditional alternatives. Paradoxically, fiat currencies and stablecoins could be more appropriate candidates for the payment infrastructure. Moreover, we also observed that NFT prices are affected by the cryptocurrency market, which highlights the risk of metaverse commerce. For future research, developers and scholars must assess the different alternatives and infrastructures that can make the metaverse a persistent reality with a proper virtual economy. However, at present, it seems that the hype has run far ahead of reality.
Evaluating the use of large language model in identifying top research questions in gastroenterology

The field of gastroenterology (GI) is constantly evolving. It is essential to pinpoint the most pressing and important research questions. To evaluate the potential of chatGPT for identifying research priorities in GI and provide a starting point for further investigation. We queried chatGPT on four key topics in GI: inflammatory bowel disease, microbiome, Artificial Intelligence in GI, and advanced endoscopy in GI. A panel of experienced gastroenterologists separately reviewed and rated the generated research questions on a scale of 1-5, with 5 being the most important and relevant to current research in GI. chatGPT generated relevant and clear research questions. Yet, the questions were not considered original by the panel of gastroenterologists. On average, the questions were rated 3.6 ± 1.4, with inter-rater reliability ranging from 0.80 to 0.98 (p < 0.001). The mean grades for relevance, clarity, specificity, and originality were 4.9 ± 0.1, 4.6 ± 0.4, 3.1 ± 0.2, 1.5 ± 0.4, respectively. Our study suggests that Large Language Models (LLMs) may be a useful tool for identifying research priorities in the field of GI, but more work is needed to improve the novelty of the generated research questions.
A Study of CNN and Transfer Learning in Medical Imaging: Advantages, Challenges, Future Scope

This paper presents a comprehensive study of Convolutional Neural Networks (CNN) and transfer learning in the context of medical imaging. Medical imaging plays a critical role in the diagnosis and treatment of diseases, and CNN-based models have demonstrated significant improvements in image analysis and classification tasks. Transfer learning, which involves reusing pre-trained CNN models, has also shown promise in addressing challenges related to small datasets and limited computational resources. This paper reviews the advantages of CNN and transfer learning in medical imaging, including improved accuracy, reduced time and resource requirements, and the ability to address class imbalances. It also discusses challenges, such as the need for large and diverse datasets, and the limited interpretability of deep learning models. What factors contribute to the success of these networks? How are they fashioned, exactly? What motivated them to build the structures that they did? Finally, the paper presents current and future research directions and opportunities, including the development of specialized architectures and the exploration of new modalities and applications for medical imaging using CNN and transfer learning techniques. Overall, the paper highlights the significant potential of CNN and transfer learning in the field of medical imaging, while also acknowledging the need for continued research and development to overcome existing challenges and limitations.
Efficient Image Retrieval via Decoupling Diffusion into Online and Offline Processing

Diffusion is commonly used as a ranking or re-ranking method in retrieval tasks to achieve higher retrieval performance, and has attracted lots of attention in recent years. A downside to diffusion is that it performs slowly in comparison to the naive k-NN search, which causes a non-trivial online computational cost on large datasets. To overcome this weakness, we propose a novel diffusion technique in this paper. In our work, instead of applying diffusion to the query, we precompute the diffusion results of each element in the database, making the online search a simple linear combination on top of the k-NN search process. Our proposed method becomes 10∼ times faster in terms of online search speed. Moreover, we propose to use late truncation instead of early truncation in previous works to achieve better retrieval performance.
Emerging Convolutions for Generative Normalizing Flows

Recently, Kingma & Dhariwal (2018) demonstrated with Glow that generative flows are capable of generating high quality images. We generalize the 1 x 1 convolutions proposed in Glow to invertible d x d convolutions, which are more flexible since they operate on both channel and spatial axes. We propose two methods to produce invertible convolutions, that have receptive fields identical to standard convolutions: Emerging convolutions are obtained by chaining specific autoregressive convolutions, and periodic convolutions are decoupled in the frequency domain. Our experiments show that the flexibility of d x d convolutions significantly improves the performance of generative flow models on galaxy images, CIFAR10 and ImageNet.
Two tributaries heterogeneous neural network based channel emulator for underwater visible light communication systems

This paper proposes a novel two tributaries heterogeneous neural network (TTHnet) based channel emulator, which is suitable for both estimating single-carrier and multi-carrier modulated channels of underwater visible light communication (UVLC). Compared to traditional neural networks, the TTHnet channel emulator has only 1932 trainable parameters, which is only 0.8% of multilayer perceptron (MLP) based channel emulator and 1% of a convolutional neural network (CNN) based channel emulator. Furthermore, it provides a more accurate estimation of the UVLC channel and greater interpretability than MLP and CNN. The experiments in this paper use carrier-less amplitude/phase modulation (CAP) and discrete multi-tone modulation (DMT) as representative examples of single-carrier and multi-carrier modulation, respectively. The experiment proves that the TTHnet based channel emulator could effectively emulate the channel response of UVLC systems both in time and frequency domain. To the best of our knowledge, this is the first time that the single-carrier and multi-carrier modulated UVLC channel is emulated by the deep neural networks based channel emulator, which will effectively accelerate the research progress of UVLC and reduce research costs of UVLC systems.
Stochastic Optimization of Sorting Networks via Continuous Relaxations

Sorting input objects is an important step in many machine learning pipelines. However, the sorting operator is non-differentiable with respect to its inputs, which prohibits end-to-end gradient-based optimization. In this work, we propose NeuralSort, a general-purpose continuous relaxation of the output of the sorting operator from permutation matrices to the set of unimodal row-stochastic matrices, where every row sums to one and has a distinct arg max. This relaxation permits straight-through optimization of any computational graph involve a sorting operation. Further, we use this relaxation to enable gradient-based stochastic optimization over the combinatorially large space of permutations by deriving a reparameterized gradient estimator for the Plackett-Luce family of distributions over permutations. We demonstrate the usefulness of our framework on three tasks that require learning semantic orderings of high-dimensional objects, including a fully differentiable, parameterized extension of the k-nearest neighbors algorithm.
SARD: Towards Scale-Aware Rotated Object Detection in Aerial Imagery

Multi-class object detection in remote sensing imagery is an important and challenging topic in computer vision. Compared with the object detection of natural scenes, remote sensing object detection has some challenges such as scale diversity, arbitrary directions and densely packed objects. To resolve these problems, this paper presents a scale-aware rotated object detection. Firstly, we propose a novel feature fusion module, which takes full advantage of high-level semantic information and low-level high resolution feature. The new feature maps are more suitable for detecting objects with a large difference in scale. Meanwhile, we design a specific weighted loss, which contains an intersection-over-union (IoU) loss and a smooth L1 loss to further address the scale diversity. Besides, in order to detect oriented and densely packed objects more accurately, we propose a normalization strategy for the representation of rotating bounding box. Our method is evaluated on two public aerial datasets DOTA and HRSC2016, and achieves competitive performances. On DOTA, we boost the mean Average Precision (mAP) to 72.95% on oriented object detection.
Robust Image Hashing Based Efficient Authentication for Smart Industrial Environment

Due to large volume and high variability of editing tools, protecting multimedia contents, and ensuring their privacy and authenticity has become an increasingly important issue in cyber-physical security of industrial environments, especially industrial surveillance. The approaches authenticating images using their principle content emerge as popular authentication techniques in industrial video surveillance applications. But maintaining a good tradeoff between perceptual robustness and discriminations is the key research challenge in image hashing approaches. In this paper, a robust image hashing method is proposed for efficient authentication of keyframes extracted from surveillance video data. A novel feature extraction strategy is employed in the proposed image hashing approach for authentication by extracting two important features: the positions of rich and nonzero low edge blocks and the dominant discrete cosine transform (DCT) coefficients of the corresponding rich edge blocks, keeping the computational cost at minimum. Extensive experiments conducted from different perspectives suggest that the proposed approach provides a trustworthy and secure way of multimedia data transmission over surveillance networks. Further, the results vindicate the suitability of our proposal for real-time authentication and embedded security in smart industrial applications compared to state-of-the-art methods.
A COVID-19 panacea in digital technologies? Challenges for democracy and higher education

Universities have transitioned to online education in order to slow the spread of COVID-19. This transition mobilizes the technological utopian imaginary that digital technologies can rescue populations from the disease. It also raises the risk of deepening neoliberal educational reforms and, by extension, poses a threat to democracy itself. This commentary explores this risk and suggests ways to resist the resulting neoliberalization of education that it could entail.
Distributed Memetic Algorithm for Outsourced Database Fragmentation

Data privacy and utility are two essential requirements in outsourced data storage. Traditional techniques for sensitive data protection, such as data encryption, affect the efficiency of data query and evaluation. By splitting attributes of sensitive associations, database fragmentation techniques can help protect data privacy and improve data utility. In this article, a distributed memetic algorithm (DMA) is proposed for enhancing database privacy and utility. A balanced best random distributed framework is designed to achieve high optimization efficiency. In order to enhance global search, a dynamic grouping recombination operator is proposed to aggregate and utilize evolutionary elements; two mutation operators, namely, merge and split, are designed to help arrange and create evolutionary elements; a two-dimension selection approach is designed based on the priority of privacy and utility. Furthermore, a splicing-driven local search strategy is embedded to introduce rare utility elements without violating constraints. Extensive experiments are carried out to verify the performance of the proposed DMA. Furthermore, the effectiveness of the proposed distributed framework and novel operators is verified.
An eight-camera fall detection system using human fall pattern recognition via machine learning by a low-cost android box

Abstract Falls are a leading cause of unintentional injuries and can result in devastating disabilities and fatalities when left undetected and not treated in time. Current detection methods have one or more of the following problems: frequent battery replacements, wearer discomfort, high costs, complicated setup, furniture occlusion, and intensive computation. In fact, all non-wearable methods fail to detect falls beyond ten meters. Here, we design a house-wide fall detection system capable of detecting stumbling, slipping, fainting, and various other types of falls at 60 m and beyond, including through transparent glasses, screens, and rain. By analyzing the fall pattern using machine learning and crafted rules via a local, low-cost single-board computer, true falls can be differentiated from daily activities and monitored through conventionally available surveillance systems. Either a multi-camera setup in one room or single cameras installed at high altitudes can avoid occlusion. This system’s flexibility enables a wide-coverage set-up, ensuring safety in senior homes, rehab centers, and nursing facilities. It can also be configured into high-precision and high-recall application to capture every single fall in high-risk zones.
Single-shot fringe projection profilometry based on deep learning and computer graphics

Multiple works have applied deep learning to fringe projection profilometry (FPP) in recent years. However, to obtain a large amount of data from actual systems for training is still a tricky problem, and moreover, the network design and optimization is still worth exploring. In this paper, we introduce graphic software to build virtual FPP systems in order to generate the desired datasets conveniently and simply. The way of constructing a virtual FPP system is described in detail firstly, and then some key factors to set the virtual FPP system much closer to reality are analyzed. With the aim of accurately estimating the depth image from only one fringe image, we also design a new loss function to enhance the overall quality and detailed information is restored. And two representative networks, U-Net and pix2pix, are compared in multiple aspects. The real experiments prove the good accuracy and generalization of the network trained by the diverse data from our virtual systems and the designed loss, providing a good guidance for real applications of deep learning methods.
ACSNet: Action-Context Separation Network for Weakly Supervised Temporal Action Localization

The object of Weakly-supervised Temporal Action Localization (WS-TAL) is to localize all action instances in an untrimmed video with only video-level supervision. Due to the lack of frame-level annotations during training, current WS-TAL methods rely on attention mechanisms to localize the foreground snippets or frames that contribute to the video-level classification task. This strategy frequently confuse context with the actual action, in the localization result. Separating action and context is a core problem for precise WS-TAL, but it is very challenging and has been largely ignored in the literature. In this paper, we introduce an Action-Context Separation Network (ACSNet) that explicitly takes into account context for accurate action localization. It consists of two branches (i.e., the Foreground-Background branch and the Action-Context branch). The Foreground-Background branch first distinguishes foreground from background within the entire video while the Action-Context branch further separates the foreground as action and context. We associate video snippets with two latent components (i.e., a positive component and a negative component), and their different combinations can effectively characterize foreground, action and context. Furthermore, we introduce extended labels with auxiliary context categories to facilitate the learning of action-context separation. Experiments on THUMOS14 and ActivityNet v1.2/v1.3 datasets demonstrate the ACSNet outperforms existing state-of-the-art WS-TAL methods by a large margin.