Reviewerly/exHarmony huggingface dataset:
Edge Real-Time Object Detection and DPU-Based Hardware Implementation for Optical Remote Sensing Images

The accuracy of current deep learning algorithms has certainly increased. However, deploying deep learning networks on edge devices with limited resources is challenging due to their inherent depth and high parameter count. Here, we proposed an improved YOLO model based on an attention mechanism and receptive field (RFA-YOLO) model, applying the MobileNeXt network as the backbone to reduce parameters and complexity, adopting the Receptive Field Block (RFB) and Efficient Channel Attention (ECA) modules to improve the detection accuracy of multi-scale and small objects. Meanwhile, an FPGA-based model deployment solution was proposed to implement parallel acceleration and low-power deployment of the detection algorithm model, which achieved real-time object detection for optical remote sensing images. We implement the proposed DPU and Vitis AI-based object detection algorithms with FPGA deployment to achieve low power consumption and real-time performance requirements. Experimental results on DIOR dataset demonstrate the effectiveness and superiority of our RFA-YOLO model for object detection algorithms. Moreover, to evaluate the performance of the proposed hardware implementation, it was implemented on a Xilinx ZCU104 board. Results of the experiments for hardware and software simulation show that our DPU-based hardware implementation are more power efficient than central processing units (CPUs) and graphics processing units (GPUs), and have the potential to be applied to onboard processing systems with limited resources and power consumption.
Metapragmatic comments deconstructing the concept of self-mockery in Chinese on social media

This paper aims to unpack perceptions of self-mockery in present-day Chinese using an online self-mockery viral event. The metapragmatic analysis of comments indicates that its interpretation is closely intertwined with other cultural factors and open to construal in a folk manner. Despite folk concepts being largely consistent with etic understandings of self-mockery, they are at the same time emotional and socially salient. The interpretation of self-mockery within a social practice is determined by how it is enacted by what person and in comparison with others, i.e., the situationally (un)approved and culturally salient meaning.
Capacity-aware fair POI recommendation combining transformer neural networks and resource allocation policy

Point of Interest (POI) recommendations have primarily focused on maximizing user satisfaction, while neglecting the needs of POIs and their operators. One such need is recommendation exposure, which can lead to envy among the POIs. Some POIs may be under-recommended, while others may be over-recommended, resulting in dissatisfaction for both staff and users due to long queues or overcrowding. Existing work has not addressed the trade-off between satisfying user preferences and being fair to POIs, which typically aim to operate at capacity. Therefore, we introduce the POI fair allocation problem to model this issue, taking into account both user satisfaction and POI exposure fairness. To address this problem, we propose a fair POI allocation technique that balances user satisfaction and POI capacity-based exposure simultaneously. Our proposed model utilizes existing (transformer neural networks and attention LSTM model) personalized POI recommendation models that capture users’ spatio-temporal influences and interests in POI visits. We then propose POI capacity-based allocation using the over-demand cut policy and under-demand add policy, which ensures POI exposure ratio and envy-freeness up to certain thresholds. We evaluate the performance of our proposed model on five datasets containing real-life POI visits. Experimental evaluations show that our proposed model outperforms baselines in terms of user and POI-based evaluation metrics. To ensure reproducibility, we have publicly shared our source code at Codeocean.
Does social media improve women's political empowerment in Africa?

The use of social media has grown significantly in Africa over the past two decades and is the subject of recent literature. In this article, we examine its direct and indirect effects on women's political empowerment (WPE) in Africa. Based on a sample of 45 African countries, we specify and estimate a panel data model using the Pooled Ordinary Least Squares (POLS) method and the System Generalized Method of Moment (S-GMM) over the period 2009–2019. Our results show that social media, as measured by the Facebook penetration rate, significantly increases WPE. Their effects are channeled through the diffusion of Information and Communication Technologies (ICT), electricity consumption, human capital and political stability. The robustness of the results is proven by alternative measures of WPE and social media. In order to strengthen WPE, public policies must increase women's access to social media.
Robust And Discriminant Local Color Pattern (RADLCP): A novel color descriptor for face recognition

In [1] Karanwal et al. introduced the novel color descriptor in Face Recognition (FR) called as Fused Local Color Pattern (FLCP). In FLCP, the RGB color format is utilized for extracting features. From R, G and B channels, the MRELBP-NI, 6 × 6 MB-LBP and RD-LBP are imposed for feature extraction and then all are integrated to form the FLCP size. FLCP beats the accuracy of various methods. The one major shortcoming observed in [1] is that the basic format RGB is used for extracting features. Literature suggests that other hybrid formats achieves better recognition rates than RGB. Motivated from literature, the proposed work uses the hybrid color space format RCrQ for feature extraction. In this format R channel is taken from RGB, Cr channel is taken from YCbCr and Q channel is taken from YIQ. On R channel, MRELBP-NI is imposed for extracting features, On Cr channel 6 × 6 MB-LBP is imposed and on Q channel RD-LBP is imposed for extracting features. Then all channel features are joined to build the robust and discriminant feature called as Robust And Discriminant Local Color Pattern (RADLCP). Compression and matching is assisted from PCA and SVMs. For evaluating results GT face dataset is used. Results proves the potency of RADLCP in contrast to gray scale based implemented descriptors. RADLCP also beats the results of FLCP. Several literature techniques are also outclassed by RADLCP. For evaluating all the results MATLAB R2021a is used.
LPMSNet: Location Pooling Multi-Scale Network for Cloud and Cloud Shadow Segmentation

Among the most difficult difficulties in contemporary satellite image-processing subjects is cloud and cloud shade segmentation. Due to substantial background noise interference, existing cloud and cloud shadow segmentation techniques would result in false detection and missing detection. We propose a Location Pooling Multi-Scale Network (LPMSNet) in this study. The residual network is utilised as the backbone in this method to acquire semantic info on various levels. Simultaneously, the Location Attention Multi-Scale Aggregation Module (LAMA) is introduced to obtain the image’s multi-scale info. The Channel Spatial Attention Module (CSA) is introduced to boost the network’s focus on segmentation goals. Finally, in view of the problem that the edge details of cloud as well as cloud shade are easily lost, this work designs the Scale Fusion Restoration Module (SFR). SFR can perform picture upsampling as well as the acquisition of edge detail information from cloud as well as cloud shade. The mean intersection over union (MIoU) accuracy of this network reached 94.36% and 81.60% on the Cloud and Cloud Shadow Dataset and the five-category dataset L8SPARCS, respectively. On the two-category HRC-WHU Dataset, the accuracy of the network on the intersection over union (IoU) reached 90.51%. In addition, in the Cloud and Cloud Shadow Dataset, our network achieves 97.17%, 96.83%, and 97.00% in precision (P), recall (R), and F1 score (F1) in cloud segmentation tasks, respectively. In the cloud shadow segmentation task, precision (P), recall (R), and F1 score (F1) reached 95.70%, 96.38%, and 96.04%, respectively. Therefore, this method has a significant advantage over the current cloud and cloud shade segmentation methods.
Attention Retractable Frequency Fusion Transformer for Image Super Resolution

Transformer-based image super-resolution (SR) has offered promising performance gains over the convolutional neural network-based one due to the adoption of parameter-independent global interactions. However, the existing Transformer-based methods are limited to obtaining enough global information due to the use of self-attention within non-overlapping windows, which restricts the receptive fields. To address this issue, we construct an effective image SR model based on the attention retractable frequency Transformer with the proposed spatial-frequency fusion block. In our method, the spatial-frequency fusion block is designed to strengthen the representation ability of the Transformer and extend the receptive field to the whole image to improve the quality of SR results. Furthermore, a progressive training strategy is proposed to use image patches with different sizes to train our SR model to further improve the SR performance. The experimental results demonstrate that our proposed method outperforms the state-of-the-art methods over various benchmark datasets, both objectively and subjectively.
Zero-Shot Action Recognition with Transformer-based Video Semantic Embedding

While video action recognition has been an active area of research for several years, zero-shot action recognition has only recently started gaining traction. In this work, we propose a novel end-to-end trained transformer model which is capable of capturing long range spatiotemporal dependencies efficiently, contrary to existing approaches which use 3D-CNNs. Moreover, to address a common ambiguity in the existing works about classes that can be considered as previously unseen, we propose a new experimentation setup that satisfies the zero-shot learning premise for action recognition by avoiding overlap between the training and testing classes. The proposed approach significantly outperforms the state of the arts in zero-shot action recognition in terms of the the top-1 accuracy on UCF-101, HMDB-51 and ActivityNet datasets.
An Effective Crop-Paste Pipeline for Few-shot Object Detection

Few-shot object detection (FSOD) aims to expand an object detector for novel categories given only a few instances for training. However, detecting novel categories with only a few samples usually leads to the problem of misclassification. In FSOD, we notice the false positive (FP) of novel categories is prominent, in which the base categories are often recognized as novel ones. To address this issue, a novel data augmentation pipeline that Crops the Novel instances and Pastes them on the selected Base images, called CNPB, is proposed. There are two key questions to be answered: (1) How to select useful base images? and (2) How to combine novel and base data? We design a multi-step selection strategy to find useful base data. Specifically, we first discover the base images which contain the FP of novel categories and select a certain amount of samples from them for the base and novel categories balance. Then the bad cases, such as the base images that have unlabeled ground truth or easily confused base instances, are removed by using CLIP. Finally, the same category strategy is adopted, in which a novel instance with category n is pasted on the base image with the FP of n. During combination, a novel instance is cropped and randomly down-sized, and thus pasted at the assigned optimal location from the randomly generated candidates in a selected base image. Our method is simple yet effective and can be easy to plug into existing FSOD methods, demonstrating significant potential for use. Extensive experiments on PASCAL VOC and MS COCO validate the effectiveness of our method.
GazeCaps: Gaze Estimation with Self-Attention-Routed Capsules

Gaze estimation is the task of estimating eye gaze from facial features. People tend to infer gaze by considering different facial properties from the whole image and their relations. However, existing methods rarely consider these various properties. In this paper, we propose a novel GazeCaps framework that represents various facial properties as different capsules. The capsules respond sensitively to transforms of facial properties by vectorial expression, which is effective for gaze estimation in which many facial components are nonlinearly transformed according to the direction of the head in addition to the perspective. Furthermore, we propose a Self-Attention Routing (SAR) module which can dynamically allocate attention to different capsules that contain important information and can be optimized as a single process without iterations. Through rigorous experiments, we confirm that the proposed method achieves state-of-the-art performance on various benchmarks. We also detail the generalization performance of the proposed model through a cross-dataset evaluation.
Ultra-Sonic Sensor based Object Detection for Autonomous Vehicles

Perception systems in autonomous vehicles (AV) have made significant advancements in recent years. Such systems leverage different sensing modalities such as cameras, LiDARs and Radars, and are powered by state-of-the-art deep learning algorithms. Ultrasonic sensors (USS) are a low-cost, durable and robust sensing technology that is particularly suitable for near-range detection in harsh weather conditions, but have received very limited attention in the perception literature. In this work, we present a novel USS-based object detection system that can enable accurate detection of objects in low-speed scenarios. The proposed pipeline involves four steps. First, the input USS data is transformed into a novel voxelized 3D point cloud leveraging the physics of USS. Next, multi-channels Bird Eye’s View (BEV) images are generated via projection operators. Later, the resolution of BEV images is enhanced by means of a rolling-window, vehicle movement-aware temporal aggregation process. Finally, the image-like data representation is used to train a deep neural network to detect and localize objects in the 2D plane. We present extensive experiments showing that the proposed framework achieves satisfactory performance across both classic and custom object detection metrics, thus bridging the usecase and literature visibility gap between USS and more established sensors.
Universal Guidance for Diffusion Models

Typical diffusion models are trained to accept a particular form of conditioning, most commonly text, and cannot be conditioned on other modalities without retraining. In this work, we propose a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain any use-specific components. We show that our algorithm successfully generates quality images with guidance functions including segmentation, face recognition, object detection, and classifier signals. Code is available at github.com/arpitbansal297/Universal-Guided-Diffusion.
Local causal structure learning for streaming features

Local causal structure learning (LCSL) is the process of discovering and distinguishing direct causes (parents) and direct effects (children) of a given target variable (T) without learning a global causal structure. However, state-of-the-art LCSL algorithms can only address static feature space, while ignoring the dynamic changes of feature space with streaming features. For high dimensional data, existing methods fail to efficiently distinguish causality between features and the target variable. To address these issues, we propose Local Causal Structure Learning for Streaming Features (LCSLSF). LCSLSF comprises two sequential steps, as it first dynamically mines causal features to construct an approximate Markov Blanket (aMB) of the target variable. It proceeds by filtering irrelevant and redundant features, retaining causal features as much as possible from streaming features. Second, it learns local causal structures by mining and splicing the V-structure of feature nodes in batches from aMB of the target variable. On benchmark Bayesian networks with the number of variables ranging from 20 to 724, LCSLSF achieves a significantly better performance than its rivals. We demonstrate its effectiveness by conducting a real-world case study on causal discovery in medical diagnostics. Code is released at https://github.com/youdianlong/LCSLSF.
Hierarchical Explanations for Video Action Recognition

To interpret deep neural networks, one main approach is to dissect the visual input and find the prototypical parts responsible for the classification. However, existing methods often ignore the hierarchical relationship between these prototypes, and thus can not explain semantic concepts at both higher level (e.g., water sports) and lower level (e.g., swimming). In this paper inspired by human cognition system, we leverage hierarchal information to deal with uncertainty. To this end, we propose HIerarchical Prototype Explainer (HIPE) to build hierarchical relations between prototypes and classes. The faithfulness of our method is verified by reducing accuracy-explainability trade-off on UCF-101 while providing multi-level explanations.
Streamlined Global and Local Features Combinator (SGLC) for High Resolution Image Dehazing

Image Dehazing aims to remove atmospheric fog or haze from an image. Although the Dehazing models have evolved a lot in recent years, few have precisely tackled the problem of High-Resolution hazy images. For this kind of image, the model needs to work on a downscaled version of the image or on cropped patches from it. In both cases, the accuracy will drop. This is primarily due to the inherent failure to combine global and local features when the image size increases. The Dehazing model requires global features to understand the general scene peculiarities and the local features to work better with fine and pixel details. In this study, we propose the Streamlined Global and Local Features Combinator (SGLC) to solve these issues and to optimize the application of any Dehazing model to High-Resolution images. The SGLC contains two successive blocks. The first is the Global Features Generator (GFG) which generates the first version of the Dehazed image containing strong global features. The second block is the Local Features Enhancer (LFE) which improves the local feature details inside the previously generated image. When tested on the Uformer architecture for Dehazing, SGLC increased the PSNR metric by a significant margin. Any other model can be incorporated inside the SGLC process to improve its efficiency on High-Resolution input data.
DIPNet: Efficiency Distillation and Iterative Pruning for Image Super-Resolution

Efficient deep learning-based approaches have achieved remarkable performance in single image super-resolution. However, recent studies on efficient super-resolution have mainly focused on reducing the number of parameters and floating-point operations through various network designs. Although these methods can decrease the number of parameters and floating-point operations, they may not necessarily reduce actual running time. To address this issue, we propose a novel multi-stage lightweight network boosting method, which can enable lightweight networks to achieve outstanding performance. Specifically, we leverage enhanced high-resolution output as additional supervision to improve the learning ability of lightweight student networks. Upon convergence of the student network, we further simplify our network structure to a more lightweight level using reparameterization techniques and iterative network pruning. Meanwhile, we adopt an effective lightweight network training strategy that combines multi-anchor distillation and progressive learning, enabling the lightweight network to achieve outstanding performance. Ultimately, our proposed method achieves the fastest inference time among all participants in the NTIRE 2023 efficient super-resolution challenge while maintaining competitive super-resolution performance. Additionally, extensive experiments are conducted to demonstrate the effectiveness of the proposed components. The results show that our approach achieves comparable performance in representative dataset DIV2K, both qualitatively and quantitatively, with faster inference and fewer number of network parameters.
Saliency-aware Stereoscopic Video Retargeting

Stereo video retargeting aims to resize an image to a desired aspect ratio. The quality of retargeted videos can be significantly impacted by the stereo video’s spatial, temporal, and disparity coherence, all of which can be impacted by the retargeting process. Due to the lack of a publicly accessible annotated dataset, there is little research on deep learning-based methods for stereo video retargeting. This paper proposes an unsupervised deep learning-based stereo video retargeting network. Our model first detects the salient objects and shifts and warps all objects such that it minimizes the distortion of the salient parts of the stereo frames. We use 1D convolution for shifting the salient objects and design a stereo video Transformer to assist the retargeting process. To train the network, we use the parallax attention mechanism to fuse the left and right views and feed the retargeted frames to a reconstruction module that reverses the retargeted frames to the input frames. Therefore, the network is trained in an unsupervised manner. Extensive qualitative and quantitative experiments and ablation studies on KITTI stereo 2012 and 2015 datasets demonstrate the efficiency of the proposed method over the existing state-of-the-art methods. The code is available at https://github.com/z65451/SVR/.
GAANet: Ghost Auto Anchor Network for Detecting Varying Size Drones in Dark

The usage of drones has tremendously increased in different sectors spanning from military to industrial applications. Despite all the benefits they offer, their misuse can lead to mishaps, and tackling them becomes more challenging particularly at night due to their small size and low visibility conditions. To overcome those limitations and improve the detection accuracy at night, we propose an object detector called Ghost Auto Anchor Network (GAANet) for infrared (IR) images. The detector uses a YOLOv5 core to address challenges in object detection for IR images, such as poor accuracy and a high false alarm rate caused by extended altitudes, poor lighting, and low image resolution. To improve performance, we implemented auto anchor calculation, modified the conventional convolution block to ghost-convolution, adjusted the input channel size, and used the AdamW optimizer. To enhance the precision of multiscale tiny object recognition, we also introduced an additional extra-small object feature extractor and detector. Experimental results in a custom IR dataset with multiple classes (birds, drones, planes, and helicopters) demonstrate that GAANet shows improvement compared to state-of-the-art detectors. In comparison to GhostNet-YOLOv5, GAANet has higher overall mean average precision (mAP@50), recall, and precision around 2.5%, 2.3%, and 1.4%, respectively. The dataset and code for this paper are available as open source at https://github.com/ZeeshanKaleem/GhostAutoAnchorNet.