Miracle-dz/UniFAR-FTUs huggingface dataset:
Quality Level Prediction of Image Compression using Block-wise Confidence-aware CNN

Almost all images are compressed to be transferred or stored, exhibiting various visual artifacts. For this reason, identifying the compression quality level with only visual cues is the starting point to enhance the image quality. This paper introduces a compression quality prediction method, named Q1Net, which yields a single quality level with over 99%-accuracy in a matter of milliseconds on mobile devices regardless of the image resolution. This real-time and high-accurate performance is attributed to the observation that most image compression methods are based upon transform coding on small blocks of different characteristics. To separately investigate and exploit the distinct visual deformations induced by one transform coding, our method measures the compression quality level on various image patches containing a basic coding block and its neighboring pixels. Our approach then elaborately selects promising candidate patches that can indicate the compression quality reliably through CNN-based statistical conﬁdence estimation. In order to make a ﬁnal decision, the proposed method fuses the prediction results from a selected number of input patches, which makes it scalable and operable on mobile devices with varying computational capabilities. According to the extensive experiments on the DIV2K dataset and an off-the-shelf smartphone, our block-wise conﬁdence-aware Q1Net achieves better performance in compression quality prediction than other well-known CNN-based methods in terms of speed and accuracy.
Learned Regularizers and Geometry for Image Denoising

Recent frameworks for image denoising have demonstrated that it can be more productive to recover an image from a smoothed version of some geometric feature of the image rather than denoise the image directly. Improvements can be found both with respect to image quality metrics as well as the preservation of fine details. The challenge in working with this data is that mathematically sound mechanisms developed for handling natural image data do not necessarily apply, and this data itself can be quite ill behaved. In this work we learn both ‘geometric’ or nonlinear higher order features and corresponding regularizers. These approaches show improvement over recent model-based deep learning (DL) image denoising methods both with respect to image quality metrics as well as the preservation of fine features. Furthermore, the proposed approach for enhancing DL architectures by incorporating geometrically-inspired features is motivated by and has the potential to feed back into mathematically sound models for solving a variety of problems in image processing.
Semi-Supervised Few-Shot Object Detection with a Teacher-Student Network

Few-shot object detection, which aims to recognize unseen objects with a few annotated instances, has attracted increasing attention in the computer vision community. Most recent works tackle this problem under the meta-learning framework based on an episodic training strategy. In this work, we advance the few-shot object detection paradigm towards a new scenario called semi-supervised few-shot object detection (SS-FSOD), where the unlabeled data are available within each episode. To address this paradigm, we propose a novel method which utilizes a dual model (teacher-student) to leverage available unlabeled data. Speciﬁcally, the teacher model provides high-quality pseudo-labels for the student model during the training process, while the student model uses the exponential moving average strategy to update the teacher model online. We also employ a two-fold correlation-guided attention module to guide RPN to generate task-speciﬁc region proposals by highlighting potential regions and informative channels. We conduct extensive experiments on three datasets MS COCO, PASCAL VOC, and FSOD. The experimental results demonstrate the effectiveness of the proposed method.
Re-ID-AR: Improved Person Re-identification in Videovia Joint Weakly Supervised Action Recognition

We uniquely consider the task of joint person re-identiﬁcation (Re-ID) and action recognition in video as a multi-task problem. In addition to the broader potential of joint Re-ID and action recognition within the context of automated multi-camera surveillance, we show that the consideration of action recognition in addition to Re-ID results in a model that learns discriminative feature representations that both improve Re-ID performance and are capable of providing viable per-view (clip-wise) action recognition. Our approach uses a single 2D Convolutional Neural Network (CNN) architecture comprising a common ResNet50-IBN backbone CNN architecture, to extract frame-level features with subsequent temporal attention for clip level feature extraction, followed by two sub-branches:- the IDentiﬁcation (sub-)Network (IDN) for person Re-ID and the Action Recognition (sub-)Network for per-view action recognition. The IDN comprises a single fully connected layer while the ARN comprises multiple attention blocks on a one-to-one ratio with the number of actions to be recognised. This is subsequently trained as a joint Re-ID and action recognition task using a combination of two task-speciﬁc, multi-loss terms via weakly labelled actions obtained over two leading benchmark Re-ID datasets (MARS, LPW). Our consideration of Re-ID and action recognition as a multi-task problem results in a multi-branch 2D CNN architecture that outperforms prior work in the ﬁeld (rank-1 (mAP) – MARS: 93 . 21% ( 87 . 23% ) , LPW: 79 . 60%) without any reliance 3D convolutions or multi-stream networks architectures as found in other contemporary work. Our work represents the ﬁrst benchmark performance for such a joint Re-ID and action recognition video understanding task, hitherto unapproached in the literature, and is accompanied by a new public dataset of supplementary action labels for the seminal MARS and LPW Re-ID datasets.
Surround-view Free Space Boundary Detection with Polar Representation

Vision-based surround-view free space detection is crucial for automatic parking assist. In this task, precise boundary localization is the most concerned problem. In this paper, we have proposed to reframe the free space as polar representation for the free space boundary, and exploit a transformer framework to regress the representation end-to-end. To restrain the overall shape of the free space, we have introduced a Triangle-IoU loss function, enabling the network to consider the boundary as a whole. Furthermore, we have proposed a challenging newly-built surround-view dataset (SVB) with boundary annotations and supplied a new metric for boundary quality. Experiments on SVB dataset validate the effectiveness of our method, which outperforms existing free space detection methods and runs in real-time with a remarkable reduction in the computational cost. Additionally, our method shows excellent generalization ability to new parking scenes.
"This Candle Has No Smell": Detecting the Effect of COVID Anosmia on Amazon Reviews Using Bayesian Vector Autoregression

While there have been many efforts to monitor or predict Covid using digital traces such as social media, one of the most distinctive and diagnostically important symptoms of Covid -- anosmia, or loss of smell -- remains elusive due to the infrequency of discussions of smell online. It was recently hypothesized that an inadvertent indicator of this key symptom may be misplaced complaints in Amazon reviews that scented products such as candles have no smell. This paper presents a novel Bayesian vector autoregression model developed to test this hypothesis, finding that "no smell" reviews do indeed reflect changes in US Covid cases even when controlling for the seasonality of those reviews. A series of robustness checks suggests that this effect is also seen in perfume reviews, but did not hold for the flu prior to Covid. These results suggest that inadvertent digital traces may be an important tool for tracking epidemics.
TinyStarGAN v2: Distilling StarGAN v2 for Efficient Diverse Image Synthesis for Multiple Domains

Image-to-image translation models, such as StarGAN v2, have enabled the translation of diverse images over multiple domains in a single framework. However, Star-GAN v2 is computationally expensive making it challenging to execute on resource-constrained environments. To reduce the computation requirement of StarGAN v2 while maintaining accuracy, we propose a novel cross distillation method that is specially designed for knowledge distillation (KD) of multiple networks in a single framework. By leveraging this new KD method, the knowledge of a multi-network large teacher Star-GAN v2 can be effectively transferred to a small student TinyStarGAN v2 framework. Without losing the quality and diversity of generated images, we reduce the size of the original framework by more than 20 × and the computation requirement by more than 5 × . Experiments on CelebA-HQ and AFHQ datasets show the effectiveness of the proposed method.
One-Shot Deep Model for End-to-End Multi-Person Activity Recognition

In this work we tackle the multi-person activity recognition problem, where actor detection, tracking, individual action recognition and group activity recognition tasks are jointly solved given an input sequence. Since related works in the literature only deal with parts of the whole problem despite sharing similar architectures, trivial combinations of them result in slow and redundant pipelines and miss the opportunity to leverage inter-task mutual dependency. This motivates us to introduce a novel deep learning model, named TrAct-Net , that can jointly solve all the above tasks in a uniﬁed architecture. A new multi-branch CNN in TrAct-Net makes inference efﬁcient and simple, and a novel relation encoder successfully takes both positional and identical relation of detections into consideration to boost both individual action and group activity recognition performances. The whole network is trained end-to-end using a multi-task learning framework. To the best of our knowledge, TrAct-Net is the ﬁrst end-to-end trainable model to solve the whole problem in a one-shot manner. Experiments on public datasets demonstrate that TrAct-Net achieves superior performance to combinations of state-of-the-arts with much fewer model parameters and faster inference speed.
Contextual Convolution Blocks

A fundamental processing layer of modern deep neural networks is the 2D convolution. It applies a ﬁlter uniformly across the input, effectively creating feature detectors that are translation invariant. In contrast, fully-connected layers are spatially selective, allowing unique detectors across the input. However, full connectivity comes at the expense of an enormous number of free parameters to be trained, the associated difﬁculty in learning without over-ﬁtting, and the loss of spatial coherence. We introduce Contextual Convolution Blocks, a novel method to create spatially selective feature detectors that are locally translation invariant. This increases the expressive power of the network beyond standard convolutional layers and allows learning unique ﬁlters for distinct regions of the input. The ﬁlters no longer need to be discriminative in regions not likely to contain the target features. This is a generalization of the Squeeze-and-Excitation architecture that introduces minimal extra parameters. We provide experimental results on three datasets and a thorough exploration into how the increased expressiveness is instantiated.
SemGIF: A Semantics Guided Incremental Few-shot Learning Framework with Generative Replay

We address the problem of incremental few-shot learning (IFSL) by leveraging the notion of generative feature replay. Learning novel concepts while preserving old knowledge is a long-lasting challenge in machine learning. The main concern in IFSL is to combat the catastrophic forgetting of the base classes whose training data are not available during the incremental stage while ensuring good generalization for the few-shot classes. Existing techniques prefer to preserve some base class samples to tackle forgetting, which does not comply with the intention of incremental learning. To this end, we propose a novel framework called Semantics Guided IFSL (SemGIF), which trains a generative model to synthesize base class samples on demand during the incremental step. Considering the importance of modeling a discriminative feature space in IFSL for separating the base and the novel classes, we propose a feature augmentation strategy where the visual embeddings are supplemented with the semantic features obtained from a word-embedding space. Such a feature space is found to produce enriched class pro-totypes to be utilized during classiﬁcation. Experimental results on CIFAR-100, CUB, mini-ImageNet, and tiered-ImageNet in the homogeneous (within-dataset) and a novel heterogeneous (cross-dataset) setup showcase sharp improvements than the literature.
Global Context and Geometric Priors for Effective Non-Local Self-Attention

Capturing relationships among local and global features in an image is crucial for proper visual understanding. However, the convolution operation is inherently limited at utilizing long-range information due to its small receptive ﬁeld. Existing approaches thus heavily rely on non-local network strategies to make up for the locality of convolutional features. Despite their successful applications in various tasks, we propose there is still considerable room for improvement, by exploring the effectiveness of global image context and position-aware representations. Notably, the concept of the relative position is surprisingly under-explored in the vision domain, whereas it has proven to be useful for modeling dependencies in machine translation tasks. In this paper, we propose a new relational reasoning module, that incorporates a contextualized diagonal matrix and 2D relative position representations. While being simple and ﬂexible, our module allows the relational representation of a feature point to encode the whole image context and its relative position information. We also explore multi-head and dropout strategies to improve the relation learning further. Extensive experiments show that our module shows consistent improvements over the state-of-the-art baselines on different vision tasks, including detection, instance segmentation, semantic segmentation, and panoptic segmentation.
Mini-batch Similarity Graphs for Robust Image Classification

Current deep learning models for image-based classiﬁcation tasks are trained using mini-batches. In the present article, we show that exploiting similarity between samples in each mini-batch can signiﬁcantly boost robustness to input perturbations, an often neglected consideration in the computer vision community. To accomplish this, we dynamically construct a similarity graph from the mini-batch samples and aggregate information using an attention module. Our experiments demonstrate an increase in robustness to local noise and black-box adversarial perturbations, when compared against a baseline model. Our approach also improves performance in diverse image-based object and scene classiﬁcation tasks, when compared against baseline models and competitive recent methods.
Crafting Object Detection in Very Low Light

Over the last decade, object detection, as a leading application in computer vision, has been intensively studied, heavily engineered and widely applicable to everyday life. However, existing object detection algorithms could easily break down under very dim environments, due to signiﬁcantly low signal-to-noise ratio (SNR). Prepending a low-light image enhancement step before detection, as a common practice, increases the computation cost substantially, yet still does not yield satisfactory results. In this paper, we systematically investigate object detection in very low light and identify several design principles that are essential to the low-light detection system. Based upon these criteria, we design a practical low-light detection system that utilizes a realistic low-light synthetic pipeline as well as an auxiliary low-light recovery module. The former can transform any labeled images from existing object detection datasets into their low-light counterparts to facilitate end-to-end training, while the latter can boost the low-light detection performance without adding additional computation cost at inference. Furthermore, we capture a real-world low-light object detection dataset, containing more than two thousand paired low/normal-light images with instance-level annotations to support this line of work. Extensive experiments collectively show the promising results of our designed detection system in very low light, paving the way for real-world object detection in the dark. Our dataset are publicly available at https://github.com/ying-fu/LODDataset .
The Contribution of Lyrics and Acoustics to Collaborative Understanding of Mood

In this work, we study the association between song lyrics and mood through a data-driven analysis. Our data set consists of nearly one million songs, with song-mood associations derived from user playlists on the Spotify streaming platform. We take advantage of state-of-the-art natural language processing models based on transformers to learn the association between the lyrics and moods. We find that a pretrained transformer-based language model in a zero-shot setting -- i.e., out of the box with no further training on our data -- is powerful for capturing song-mood associations. Moreover, we illustrate that training on song-mood associations results in a highly accurate model that predicts these associations for unseen songs. Furthermore, by comparing the prediction of a model using lyrics with one using acoustic features, we observe that the relative importance of lyrics for mood prediction in comparison with acoustics depends on the specific mood. Finally, we verify if the models are capturing the same information about lyrics and acoustics as humans through an annotation task where we obtain human judgments of mood-song relevance based on lyrics and acoustics.
Coloured Edge Maps for Oil Palm Ripeness Classification

The task of grading oil palm bunches by ripeness poses a number of signiﬁcant challenges for computer vision. The small difference in hue between ripe and unripe bunches means that colour-based models are susceptible to errors when presented with images shot in novel lighting conditions. In this paper, we investigate the effectiveness and performance characteristics of coloured edge maps when used as an input feature to a Convolutional Neural Network (CNN) by comparing the Laplacian of Gaussian, Sobel, Prewitt, and Kirsch edge extraction techniques. We show that under normal lighting conditions, coloured edge maps are able to match the performance of fully-coloured images. More notably, they signiﬁcantly outperform fully-coloured images when variance to lighting is applied. When images are darkened or brightened, classiﬁcation accuracy for fully-coloured images drops by 19.89% vs only 4.97% on average for the coloured edge map methods tested. This is of major beneﬁt in commercial applications, where images are often captured by a multitude of devices under different lighting conditions, leading to potentially unreliable performance when fully-coloured images are used. The code used for this paper will be made available at https://github.com/weiyuen.
Community Under Surveillance: Impacts of Marginalization on an Online Labor Forum

A breadth of literature has examined how gig workers use online forums. The past literature focuses primarily on how gig workers for mainstream corporate platforms leverage forums. Yet, marginalization, stigma, censorship, and criminalization all shape how people, including gig workers, use digital technology. In this work, we seek to take a first step toward understanding how marginalized, stigmatized, digitally censored, and in some cases criminalized gig workers leverage the affordances of online public forums to build community and increase their welfare. To do so, we conduct a qualitative analysis of 4,000 posts and 25,851 comments shared over four months in a large online public forum used by sex workers for peer support. Sex workers sit at the intersection of multiple marginalized communities, and thus offer a lens into the broader use of forums by marginalized labor communities. Our findings offer insight into how these workers use moderation to preserve their safety and community. Further, we highlight similarities and differences between how this community utilizes the forum platform and prior scholarship on the use of online forums by gig workers.
From Seq2Seq Recognition to Handwritten Word Embeddings

In this work, we propose a system for automatically extracting handwritten word embeddings, using the encoding module of a Sequence-to-Sequence (Seq2Seq) recognition network. These embeddings are proven to be very discriminative, since they can be effectively used for Keyword Spotting, while they can also be fully decoded into the target string following the Seq2Seq rationale. Architecture-wise, the proposed system incor-porates several novel modules (e.g. auto-encoder path or non-recurrent CTC-branch) that assist the training procedure and boost performance. Additionally, we also show how to further process these embeddings/representations with a binarization scheme to provide compact and highly efﬁcient descriptors, suitable for Keyword Spotting. Numerical results validate the usefulness of the proposed architecture, as our method provides state-of-the-art results for both recognition and spotting.
Design and development of printed acoustic sensor

The objective of this work was to design and fabricate a low cost resonant capacitive acoustic sensor using printing techniques. It falls within the frame of a collaborative research project named “Spinnaker”, set up by TAGSYS RFID, a French company, which has planned to integrate this sensor to improve the geolocalization of their RFID tags. This work started with the design and optimization of the sensor using finite element modeling (COMSOL) and design of experiments (DOE). This first step has enabled the identification of the optimum set of parameters and demonstrated that the output responses were in accordance with the specifications. Then, we have developed the different technological building blocks required for the fabrication of the prototypes using jointly the 2D inkjet printing technique and 3D printing method. The functionality of the sensors has been characterized using both capacitive and acoustic measurements using laser Doppler vibrometer. Experimental results showed that sensitivity and selectivity were within the specifications and in good agreement with the modeling results. Finally, we investigated the piezoelectric approach which could be an interesting option to the capacitive one. Since no inkjet printable piezoelectric ink is commercially available, stable inkjet printable polyvinylidene fluoride-trifluoroethylene (PVDF-TrFE) ink has been developed. PVDF-TrFE layers were then successfully printed and characterized. The results were quite promising, however further improvements of the ink and printing process are required before stepping towards piezoelectric based device fabrication.