Reviewerly/exHarmony huggingface dataset:
Lost in Compression: the Impact of Lossy Image Compression on Variable Size Object Detection within Infrared Imagery

Lossy image compression strategies allow for more efficient storage and transmission of data by encoding data to a reduced form. This is essential enable training with larger datasets on less storage-equipped environments. However, such compression can cause severe decline in performance of deep Convolution Neural Network (CNN) architectures even when mild compression is applied and the resulting compressed imagery is visually identical. In this work, we apply the lossy JPEG compression method with six discrete levels of increasing compression {95, 75, 50, 15, 10, 5} to infrared band (thermal) imagery. Our study quantitatively evaluates the affect that increasing levels of lossy compression has upon the performance of characteristically diverse object detection architectures (Cascade-RCNN, FSAF and Deformable DETR) with respect to varying sizes of objects present in the dataset. When training and evaluating on uncompressed data as a baseline, we achieve maximal mean Average Precision (mAP) of 0.823 with Cascade R-CNN across the FLIR dataset, outperforming prior work. The impact of the lossy compression is more extreme at higher compression levels (15, 10, 5) across all three CNN architectures. However, re-training models on lossy compressed imagery notably ameliorated performances for all three CNN models with an average increment of ∼ 76% (at higher compression level 5). Additionally, we demonstrate the relative sensitivity of differing object areas {tiny, small, medium, large} with respect to the compression level. We show that tiny and small objects are more sensitive to compression than medium and large objects. Overall, Cascade R-CNN attains the maximal mAP across most of the object area categories.
Probabilistic Compositional Embeddings for Multimodal Image Retrieval

Existing works in image retrieval often consider retrieving images with one or two query inputs, which do not generalize to multiple queries. In this work, we investigate a more challenging scenario for composing multiple multi-modal queries in image retrieval. Given an arbitrary number of query images and (or) texts, our goal is to retrieve target images containing the semantic concepts specified in multiple multimodal queries. To learn an informative embedding that can flexibly encode the semantics of various queries, we propose a novel multimodal probabilistic composer (MPC). Specifically, we model input images and texts as probabilistic embeddings, which can be further composed by a probabilistic composition rule to facilitate image retrieval with multiple multimodal queries. We propose a new benchmark based on the MS-COCO dataset and evaluate our model on various setups that compose multiple images and (or) text queries for multimodal image retrieval. Without bells and whistles, we show that our probabilistic model formulation significantly outperforms existing related methods on multimodal image retrieval while generalizing well to query with different amounts of inputs given in arbitrary visual and (or) textual modalities. Code is here: https://github.com/andreineculai/MPC.
Pass Receiver Prediction in Soccer using Video and Players’ Trajectories

In soccer, passing is one of the most fundamental actions for building tactics. Automatic prediction of the pass receiver can be useful in many situations, such as in player and team analysis and entertainment. In previous studies, the prediction is based on tracking data, in particular, time-series data of the two-dimensional positions of the players on the field, and little use has been made of video information such as the players’ own posture and facial orientation. Thus, this paper aims to build a pass receiver prediction model that combines visual information with the trajectories of the players and the ball. We extract the features of the players’ body movements from the video and the features of their movements on the field from the trajectories by using 3D convolutional networks and long short-term memory and learn the interactions between each player by using a transformer. Our study evaluation used wide-angle video and tracking data of 20 players, i.e., all players on the field excluding the goalkeepers. The results show that the prediction accuracy is greatly improved by using the video information.
Pose-based Contrastive Learning for Domain Agnostic Activity Representations

While recognition accuracies of video classification models trained on conventional benchmarks are gradually saturating, recent studies raise alarm about the learned representations not generalizing well across different domains. Learning abstract concepts behind an activity instead of overfitting to the appearances and biases of a specific benchmark domain is vital for building generalizable behaviour understanding models. In this paper, we introduce Pose-based High Level View Contrasting (P-HLVC), a novel method that leverages human pose dynamics as a supervision signal aimed at learning domain-invariant activity representations. Our model learns to link image sequences to more abstract body pose information through iterative contrastive clustering and the Sinkhorn-Knopp algorithm, providing us with video representations more resistant to domain shifts. We demonstrate the effectiveness of our approach in a cross-domain action recognition setting and achieve significant improvements on the synthetic-to-real Sims4Action benchmark. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Entangled microwave photons generation using cryogenic low noise amplifier (transistor nonlinearity effects)

Abstract This article mainly focuses on important quantum phenomenon called entanglement arising the nonlinearity property. This study uses a unique approach in which transistor nonlinearity effect (third-order nonlinearity) entangled microwave photons are created in a cryogenic low-noise amplifier (LNA). For entanglement analysis, the Hamiltonian of the designed cryogenic LNA (containing two coupled oscillators) is derived, and then, using the dynamic equation of motion, the oscillator’s number of photons and the phase-sensitive cross-correlation factor are calculated in the Fourier domain to calculate the entanglement metric. The oscillators are coupled to each other through the gate–drain capacitor, and nonlinear transconductance is as an important factor strongly manipulating the entanglement. As a main conclusion, the study shows that the designed circuit using transistor third-order nonlinearity has the ability to generate the entangled microwave photons at very low intrinsic transconductance and more importantly when the noise figure (NF) is strongly minimized. As a complementary task, the printed circuit board of the cryogenic LNA is designed and simulated to verify the ability of the circuit to achieve an ultralow NF, by which the probability of the generation of entangled microwave photons is increased.
Dust InSMS: Intelligent soiling measurement system for dust detection on solar mirrors using computer vision methods

The dust accumulation strongly impacts the optical efficiency of solar concentrators, in particular the reflectivity of solar mirrors. Therefore, reducing the impact of reflectivity losses due to soiling and optimizing cleaning strategy are key factors. In this paper, the impact of dust accumulation on the reflectivity parameter of Fresnel mirrors is studied at the GEP research platform during the dry period. Based on the collected data, a new system for dust detection is proposed based on the classification approach using the convolutional neural networks and image processing algorithms in which no similar work is presented in the literature that uses the same approach to quantify the soiling phenomenon on CSP mirrors. The test loss and accuracy obtained by the proposed model are respectively 0.28 and 0.96. The outdoor validation results obtained so far suggest that the Dust InSMS concept and method could be a promising efficient and low-cost sensor. As the proposed system performs the GPS coordinates for each measurement, an optimal cleaning scenario is developed based on genetic algorithms to optimize the cleaning scenario and to come up with the shortest cleaning path.
NeuralAnnot: Neural Annotator for 3D Human Mesh Training Sets

Most 3D human mesh regressors are fully supervised with 3D pseudo-GT human model parameters and weakly supervised with GT 2D/3D joint coordinates as the 3D pseudo-GTs bring great performance gain. The 3D pseudo-GTs are obtained by annotators, systems that iteratively fit 3D human model parameters to GT 2D/3D joint coordinates of training sets in the pre-processing stage of the regressors. The fitted 3D parameters at the last fitting iteration become the 3D pseudo-GTs, used to fully super-vise the regressors. Optimization-based annotators, such as SMPLify-X, have been widely used to obtain the 3D pseudo-GTs. However, they often produce wrong 3D pseudo-GTs as they fit the 3D parameters to GT of each sample independently. To overcome the limitation, we present NeuralAnnot, a neural network-based annotator. The main idea of NeuralAnnot is to employ a neural network-based regressor and dedicate it for the annotation. Assuming no 3D pseudo-GTs are available, NeuralAnnot is weakly supervised with GT 2D/3D joint coordinates of training sets. The testing results on the same training sets become 3D pseudo-GTs, used to fully supervise the regressors. We show that 3D pseudo-GTs of NeuralAnnot are highly beneficial to train the regressors. We made our 3D pseudo-GTs publicly available.
A Multi-granularity Retrieval System for Natural Language-based Vehicle Retrieval

We focus on the task of the Natural language-based vehicle track retrieval of the 6th AI City Challenge. Performing target vehicle retrieval using natural language descriptions is a comprehensive task, requiring a model to first understand the semantics of the language and vision modalities and then match them to generate accurate retrieval results. However, this task involves the following challenges: (1) the ambiguity of the natural language descriptions towards a target vehicle; (2) the matching between the linguistic semantics of the language descriptions and the corresponding static and dynamic properties of the target vehicle; (3) the shortage of the annotated language and target vehicle pairs. Obviously, focusing on solving a subset of the problems cannot generate a robust retrieval model. Therefore, we propose a multi-granularity retrieval system to solve this task, consisting of three main modules: (1) Language parsing module that aims to obtain the fine-grained vehicle attributes (e.g. color, type and motion) from the language descriptions; (2) Language-augmented multi-query vehicle track retrieval module that serves as our baseline model to incorporate information from multiple imperfect queries; (3) Target vehicle attributes enhancement module that explicitly fuses the static and dynamic properties of the target vehicle to generate the final retrieval results. Our system has achieved the 1st place on the 6th AI City Challenge, yielding a strong performance on the private test set.
ViTOL: Vision Transformer for Weakly Supervised Object Localization

Weakly supervised object localization (WSOL) aims at predicting object locations in an image using only image-level category labels. Common challenges that image classification models encounter when localizing objects are, (a) they tend to look at the most discriminative features in an image that confines the localization map to a very small region, (b) the localization maps are class agnostic, and the models highlight objects of multiple classes in the same image and, (c) the localization performance is affected by background noise. To alleviate the above challenges we introduce the following simple changes through our proposed method ViTOL. We leverage the vision-based transformer for self-attention and introduce a patch-based attention dropout layer (p-ADL) to increase the coverage of the localization map and a gradient attention rollout mechanism to generate class-dependent attention maps. We conduct extensive quantitative, qualitative and ablation experiments on the ImageNet-1K and CUB datasets. We achieve state-of-the-art MaxBoxAcc-V2 localization scores of 70.47% and 73.17% on the two datasets respectively. Code is available on https://github.com/Saurav-31/ViTOL.
Efficient tracking of team sport players with few game-specific annotations

One of the requirements for team sports analysis is to track and recognize players. Many tracking and re-identification methods have been proposed in the context of video surveillance. They show very convincing results when tested on public datasets such as the MOT challenge. However, the performance of these methods are not as satisfactory when applied to player tracking. Indeed, in addition to moving very quickly and often being occluded, the players wear the same jersey, which makes the task of re-identification very complex. Some recent tracking methods have been developed more specifically for the team sport context. Due to the lack of public data, these methods use private datasets that make impossible a comparison with them. In this paper, we propose a new generic method to track team sport players during a full game thanks to few human annotations collected via a semi-interactive system. Non-ambiguous tracklets and their appearance features are automatically generated with a detection and a re-identification network both pre-trained on public datasets. Then an incremental learning mechanism trains a Transformer to classify identities using few game-specific human annotations. Finally, tracklets are linked by an association algorithm. We demonstrate the efficiency of our approach on a challenging rugby sevens dataset. To overcome the lack of public sports tracking dataset, we publicly release this dataset at https://kalisteo.cea.fr/index.php/free-resources/. We also show that our method is able to track rugby sevens players during a full match, if they are observable at a minimal resolution, with the annotation of only 6 few seconds length tracklets per player.
A robust non-blind deblurring method using deep denoiser prior

The existing non-blind deblurring methods are mostly susceptible to noise in the given blurring kernel, which is usually estimated from the observed image. This will produce undesirable ringing artifacts around the recovered edges when the given kernel is not accurate enough. Besides, the noise and outliers in the observed images may also severely degrade the performance of the deblurring methods. Considering these factors, we designed a robust non-blind deblurring method taking all these noises into account. In this paper, we propose a kernel error term to rectify the given kernel in the midst of the deconvolution process. A residual error term is also introduced to deal with the outliers caused by noise or saturation. A deep learning denoiser prior is adopted to reserve the fine textures in the recovered image. The experiments show clearly that the proposed method achieves remarkable progress in both the visual quality and the numerical results of the recovered images compared to the state-of-the-art deblurring methods.
PRDECT-ID: Indonesian product reviews dataset for emotions classification tasks

Recognizing emotions is vital in communication. Emotions convey additional meanings to the communication process. Nowadays, people can communicate their emotions on many platforms; one is the product review. Product reviews in the online platform are an important element that affects customers' buying decisions. Hence, it is essential to recognize emotions from the product reviews. Emotions recognition from the product reviews can be done automatically using a machine or deep learning algorithm. Dataset can be considered as the fuel to model the recognizer. However, only a limited dataset exists in recognizing emotions from the product reviews, particularly in a local language. This research contributes to the dataset collection of 5400 product reviews in Indonesian. It was carefully curated from various (29) product categories, annotated with five emotions, and verified by an expert in clinical psychology. The dataset supports an innovative process to build automatic emotion classification on product reviews.
A unified total variation method for underwater image enhancement

Underwater images usually suffer from color casts and low contrast due to the absorption and scattering of light by the water medium. The degradation is caused not only by the light attenuation on the scene-sensor path but also by the light attenuation on the water surface-scene path. To eliminate the dual-path light attenuation, we propose a novel unified total variation method based on an extended underwater imaging model. Unlike previous variation-based methods that only consider light propagation along the scene-sensor path, we additionally include light propagation along the water surface-scene path in the underwater imaging model. In the proposed variational framework, we transform underwater image enhancement into two subproblems and construct different prior knowledge-guided optimization functions for them. The two subproblems aim to remove the light attenuation along the scene-sensor and surface-scene paths. Moreover, we present an alternating direction minimization algorithm based on an augmented Lagrange multiplier to address the optimization problems. The subjective and objective experimental results on underwater images with different attenuation characteristics demonstrate that the proposed method achieves good performance in underwater image enhancement.
Improved GNN method with finite-time convergence for time-varying Lyapunov equation

Dynamic neural networks are efficient for solving algebraic equations. Among them, the gradient neural network (GNN) has the lowest model complexity. The conventional GNN models have exponential convergence when dealing with static Lyapunov equations (LEs), but fail to find the accurate solution when the solutions change with time. To fill this gap, in this paper, we propose an improved GNN method by introducing additional nonlinearity into a traditional GNN model for handling time-varying LEs. It is shown that the proposed improved GNN (IGNN) method has finite-time convergence when it is applied to time-varying LEs, even if there are bounded additive time-varying noises. Simulation results demonstrate the efficacy and advantages of the proposed method over the existing GNN methods and other dynamic neural network methods for time-varying LEs solving.
Packaging and Interconnect Considerations in Neuromorphic Photonic Accelerators

Developing compute platforms capable of performing computations at high speed is essential for data processing in the next generation of data centers and edge devices. A neuromorphic photonic accelerator on a silicon photonic platform is a promising solution. Compared to silicon photonic data communication transceiver modules, neuromorphic photonic accelerators constitute a large number of active and passive components and optoelectronic devices to handle the parallel processing. Thus, an increased number of optical and electrical interconnects are required, making the packaging of such processors challenging. Moreover, thermal and electrical crosstalk can dramatically degrade the performance of such processors. Thus, packaging a neuromorphic photonic accelerator for efficient processing and data movement requires careful considerations at the chip, module, and board levels. This work investigates the challenges and potential solutions for optical coupling, optical and electrical interconnections, processor-memory communication, and thermal and electrical cross-talk to develop neuromorphic photonic accelerators.
SLAM: SLO-Aware Memory Optimization for Serverless Applications

Serverless computing paradigm has become more ingrained into the industry, as it offers a cheap alternative for application development and deployment. This new paradigm has also created new kinds of problems for the developer, who needs to tune memory configurations for balancing cost and performance. Many researchers have addressed the issue of minimizing cost and meeting Service Level Objective (SLO) requirements for a single FaaS function, but there has been a gap for solving the same problem for an application consisting of many FaaS functions, creating complex application workflows.In this work, we designed a tool called SLAM to address the issue. SLAM uses distributed tracing to detect the relationship among the FaaS functions within a serverless application. By modeling each of them, it estimates the execution time for the application at different memory configurations. Using these estimations, SLAM determines the optimal memory configuration for the given serverless application based on the specified SLO requirements and user-specified objectives (minimum cost or minimum execution time). We demonstrate the functionality of SLAM on AWS Lambda by testing on four applications. Our results show that the suggested memory configurations guarantee that more than 95% of requests are completed within the predefined SLOs.
Self-Supervised Monocular Depth Estimation Using Hybrid Transformer Encoder

Depth estimation using monocular camera sensors is an important technique in computer vision.Supervised monocular depth estimation requires a lot of data acquired from depth sensors.However, acquiring depth data is an expensive task.We sometimes cannot acquire data due to the limitations of the sensor.View synthesis-based depth estimation research is a self-supervised learning method that does not require depth data supervision.Previous studies mainly use the convolutional neural network (CNN)-based networks in encoders.The CNN is suitable for extracting local features through convolution operation.Recent vision transformers (ViTs) are suitable for global feature extraction based on multiself-attention modules.In this article, we propose a hybrid network combining the CNN and ViT networks in self-supervised learning-based monocular depth estimation.We design an encoder-decoder structure that uses CNNs in the earlier stage of extracting local features and a ViT in the later stages of extracting global features.We evaluate the proposed network through various experiments based on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) and Cityscapes datasets.The results showed higher performance than previous studies and reduced parameters and computations.Codes and trained models are available at https://github.com/fogfog2/manydepthformer.18