Reviewerly/exHarmony huggingface dataset:
MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge

Large scale Vision Language (VL) models have shown tremendous success in aligning representations between visual and text modalities. This enables remarkable progress in zero-shot recognition, image generation & editing, and many other exciting tasks. However, VL models tend to over-represent objects while paying much less attention to verbs, and require additional tuning on video data for best zero-shot action recognition performance. While previous work relied on large-scale, fully-annotated data, in this work we propose an unsupervised approach. We adapt a VL model for zero-shot and few-shot action recognition using a collection of unlabeled videos and an unpaired action dictionary. Based on that, we leverage Large Language Models and VL models to build a text bag for each unlabeled video via matching, text expansion and captioning. We use those bags in a Multiple Instance Learning setup to adapt an image-text backbone to video data. Although finetuned on unlabeled video data, our resulting models demonstrate high transferability to numerous unseen zero-shot downstream tasks, improving the base VL model performance by up to 14%, and even comparing favorably to fully-supervised baselines in both zero-shot and few-shot video recognition transfer. The code is released at https://github.com/wlin-at/MAXI.
Global Features are All You Need for Image Retrieval and Reranking

Image retrieval systems conventionally use a two-stage paradigm, leveraging global features for initial retrieval and local features for reranking. However, the scalability of this method is often limited due to the significant storage and computation cost incurred by local feature matching in the reranking stage. In this paper, we present SuperGlobal, a novel approach that exclusively employs global features for both stages, improving efficiency without sacrificing accuracy. SuperGlobal introduces key enhancements to the retrieval system, specifically focusing on the global feature extraction and reranking processes. For extraction, we identify sub-optimal performance when the widely-used ArcFace loss and Generalized Mean (GeM) pooling methods are combined and propose several new modules to improve GeM pooling. In the reranking stage, we introduce a novel method to update the global features of the query and top-ranked images by only considering feature refinement with a small set of images, thus being very compute and memory efficient. Our experiments demonstrate substantial improvements compared to the state of the art in standard benchmarks. Notably, on the Revisited Oxford+1M Hard dataset, our single-stage results improve by 7.1%, while our two-stage gain reaches 3.7% with a strong 64,865× speedup. Our two-stage system surpasses the current single-stage state-of-the-art by 16.3%, offering a scalable, accurate alternative for high-performing image retrieval systems with minimal time overhead.Code: https://github.com/ShihaoShao-GH/SuperGlobal.
Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models

Token-based masked generative models are gaining popularity for their fast inference time with parallel decoding. While recent token-based approaches achieve competitive performance to diffusion-based models, their generation performance is still suboptimal as they sample multiple tokens simultaneously without considering the dependence among them. We empirically investigate this problem and propose a learnable sampling model, Text-Conditioned Token Selection (TCTS), to select optimal tokens via localized supervision with text information. TCTS improves not only the image quality but also the semantic alignment of the generated images with the given texts. To further improve the image quality, we introduce a cohesive sampling strategy, Frequency Adaptive Sampling (FAS), to each group of tokens divided according to the self-attention maps. We validate the efficacy of TCTS combined with FAS with various generative tasks, demonstrating that it significantly outperforms the baselines in image-text alignment and image quality. Our text-conditioned sampling framework further reduces the original inference time by more than 50 % without modifying the original generative model.
Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning

Benefiting from prompt tuning, recent years have witnessed the promising performance of pre-trained vision-language models, e.g., CLIP, on versatile downstream tasks. In this paper, we focus on a particular setting of learning adaptive prompts on the fly for each test sample from an unseen new domain, which is known as test-time prompt tuning (TPT). Existing TPT methods typically rely on data augmentation and confidence selection. However, conventional data augmentation techniques, e.g., random resized crops, suffers from the lack of data diversity, while entropy-based confidence selection alone is not sufficient to guarantee prediction fidelity. To address these issues, we propose a novel TPT method, named DiffTPT, which leverages pre-trained diffusion models to generate diverse and informative new data. Specifically, we incorporate augmented data by both conventional method and pre-trained stable diffusion to exploit their respective merits, improving the model’s ability to adapt to unknown new test data. Moreover, to ensure the prediction fidelity of generated data, we introduce a cosine similarity-based filtration technique to select the generated data with higher similarity to the single test sample. Our experiments on test datasets with distribution shifts and unseen categories demonstrate that DiffTPT improves the zero-shot accuracy by an average of 5.13% compared to the state-of-the-art TPT method.
Continuously Masked Transformer for Image Inpainting

A novel continuous-mask-aware transformer for image inpainting, called CMT, is proposed in this paper, which uses a continuous mask to represent the amounts of errors in tokens. First, we initialize a mask and use it during the self-attention. To facilitate the masked self-attention, we also introduce the notion of overlapping tokens. Second, we update the mask by modeling the error propagation during the masked self-attention. Through several masked self-attention and mask update (MSAU) layers, we predict initial inpainting results. Finally, we refine the initial results to reconstruct a more faithful image. Experimental results on multiple datasets show that the proposed CMT algorithm outperforms existing algorithms significantly. The source codes are available at https://github.com/keunsoo-ko/CMT.
DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion

We present DreamPose, a diffusion-based method for generating animated fashion videos from still images. Given an image and a sequence of human body poses, our method synthesizes a video containing both human and fabric motion. To achieve this, we transform a pre-trained text-to-image model (Stable Diffusion [16]) into a pose-and-image guided video synthesis model, using a novel finetuning strategy, a set of architectural changes to support the added conditioning signals, and techniques to encourage temporal consistency. We fine-tune on a collection of fashion videos from the UBC Fashion dataset [50]. We evaluate our method on a variety of clothing styles and poses, and demonstrate that our method produces state-of-the-art results on fashion video animation. Video results are available on our project page: https://grail.cs.washington.edu/projects/dreampose
Perpetual Humanoid Control for Real-time Simulated Avatars

We present a physics-based humanoid controller that achieves high-fidelity motion imitation and fault-tolerant behavior in the presence of noisy input (e.g. pose estimates from video or generated from language) and unexpected falls. Our controller scales up to learning ten thousand motion clips without using any external stabilizing forces and learns to naturally recover from fail-state. Given reference motion, our controller can perpetually control simulated avatars without requiring resets. At its core, we propose the progressive multiplicative control policy (PMCP), which dynamically allocates new network capacity to learn harder and harder motion sequences. PMCP allows efficient scaling for learning from large-scale motion databases and adding new tasks, such as fail-state recovery, without catastrophic forgetting. We demonstrate the effectiveness of our controller by using it to imitate noisy poses from video-based pose estimators and language-based motion generators in a live and real-time multi-person avatar use case.
Zero-guidance Segmentation Using Zero Segment Labels

The joint visual-language model CLIP has enabled new and exciting applications, such as open-vocabulary segmentation, which can locate any segment given an arbitrary text query. In our research, we ask whether it is possible to discover semantic segments without any user guidance in the form of text queries or predefined classes, and label them using natural language automatically? We propose a novel problem zero-guidance segmentation and the first baseline that leverages two pre-trained generalist models, DINO and CLIP, to solve this problem without any fine-tuning or segmentation dataset. The general idea is to first segment an image into small over-segments, encode them into CLIP’s visual-language space, translate them into text labels, and merge semantically similar segments together. The key challenge, however, is how to encode a visual segment into a segment-specific embedding that balances global and local context information, both useful for recognition. Our main contribution is a novel attention-masking technique that balances the two contexts by analyzing the attention layers inside CLIP. We also introduce several metrics for the evaluation of this new task. With CLIP’s innate knowledge, our method can precisely locate the Mona Lisa painting among a museum crowd (Figure 1). More results are available at https://zero-guide-seg.github.io/.
Adaptive Superpixel for Active Learning in Semantic Segmentation

Learning semantic segmentation requires pixel-wise annotations, which can be time-consuming and expensive. To reduce the annotation cost, we propose a superpixel-based active learning (AL) framework, which collects a dominant label per superpixel instead. To be specific, it consists of adaptive superpixel and sieving mechanisms, fully dedicated to AL. At each round of AL, we adaptively merge neighboring pixels of similar learned features into superpixels. We then query a selected subset of these superpixels using an acquisition function assuming no uniform superpixel size. This approach is more efficient than existing methods, which rely only on innate features such as RGB color and assume uniform superpixel sizes. Obtaining a dominant label per superpixel drastically reduces annotators’ burden as it requires fewer clicks. However, it inevitably introduces noisy annotations due to mismatches between superpixel and ground truth segmentation. To address this issue, we further devise a sieving mechanism that identifies and excludes potentially noisy annotations from learning. Our experiments on both Cityscapes and PASCAL VOC datasets demonstrate the efficacy of adaptive superpixel and sieving mechanisms.
DS-Fusion: Artistic Typography via Discriminated and Stylized Diffusion

We introduce a novel method to automatically generate an artistic typography by stylizing one or more letter fonts to visually convey the semantics of an input word, while ensuring that the output remains readable. To address an assortment of challenges with our task at hand including conflicting goals (artistic stylization vs. legibility), lack of ground truth, and immense search space, our approach utilizes large language models to bridge texts and visual images for stylization and build an unsupervised generative model with a diffusion model backbone. Specifically, we employ the denoising generator in Latent Diffusion Model (LDM), with the key addition of a CNN-based discriminator to adapt the input style onto the input text. The discriminator uses rasterized images of a given letter/word font as real samples and the output of the denoising generator as fake samples. Our model is coined DS-Fusion for discriminated and stylized diffusion. We showcase the quality and versatility of our method through numerous examples, qualitative and quantitative evaluation, and ablation studies. User studies comparing to strong baselines including CLIPDraw, DALL-E 2, Stable Diffusion, as well as artist-crafted typographies, demonstrate strong performance of DS-Fusion. Code is available at https://ds-fusion.github.io/.
A Comprehensive Review On Two-Stage Object Detection Algorithms

Two-stage object detection algorithms have gained significant attention in computer vision due to their robustness and accuracy in detecting objects in images. These algorithms consist of two main stages: region proposal generation and object classification/localization. The region proposal stage efficiently generates a diverse set of candidate object regions using various techniques, narrowing down the search space. In the second stage, deep convolutional neural networks extract discriminative features from the proposed regions, enabling precise object classification and localization. These algorithms have demonstrated impressive performance in complex scenes with occlusions, scale variations, and multiple objects. However, they have limitations such as slower inference speeds, increased computational complexity, and the need for additional training steps. To address these limitations, more efficient algorithms like single-stage detectors have emerged. Two-stage object detection algorithms offer a powerful framework for accurate and robust object detection, and ongoing research aims to refine and improve these algorithms, expanding the capabilities of computer vision applications.
SuS-X: Training-Free Name-Only Transfer of Vision-Language Models

Contrastive Language-Image Pre-training (CLIP) has emerged as a simple yet effective way to train large-scale vision-language models. CLIP demonstrates impressive zero-shot classification and retrieval performance on diverse downstream tasks. However, to leverage its full potential, fine-tuning still appears to be necessary. Fine-tuning the entire CLIP model can be resource-intensive and unstable. Moreover, recent methods that aim to circumvent this need for fine-tuning still require access to images from the target task distribution. In this paper, we pursue a different approach and explore the regime of training-free "name-only transfer" in which the only knowledge we possess about the downstream task comprises the names of downstream target categories. We propose a novel method, SuS-X, consisting of two key building blocks— "SuS" and "TIP-X", that requires neither intensive fine-tuning nor costly labelled data. SuS-X achieves state-of-the-art (SoTA) zero-shot classification results on 19 benchmark datasets. We further show the utility of TIP-X in the training-free few-shot setting, where we again achieve SoTA results over strong training-free baselines. Code is available at https://github.com/vishaal27/SuS-X.
Metamorphosing Teaching and Learning with Python: A Practical Guide for Teachers Employing Text Data Analysis Using the Bag of Words (BoW) Model

This research explores the application of the Bag of Words (BoW) model for text data analysis with the goal of enhancing teaching-learning in the classroom. The BoW model expresses text as a “bag” of its individual words while ignoring word order. The study covers the use of the BoW model toward the creation of intelligent teaching systems. These systems identify areas where students may be having difficulty and offer focused feedback and guidance by examining the language used by students as they work through an issue or subject. The paper uses snippets of Python code as examples and their outputs to demonstrate how the BoW model can be used to improve learning outcomes. The BoW model’s importance in assessing educational resources, developing intelligent tutoring programmes, and analysing student writing is greatly emphasised in the paper.
Robustness Analysis uncovers Language Proficiency Bias in Emotion Recognition Systems

Emotion recognition in conversations (ERC) has rapidly emerged as a vital instrument in enhancing human-computer interactions. However, concerns about the fairness and biases of these ERC systems persist and remain to be addressed by assessing their robustness. This study presents a methodology to analyze the robustness and bias of an ERC system by including complexities of user input with varying English language proficiency. We develop a novel, hybrid approach to create text perturbations by combining natural language generation techniques with rule-based constraints to simulate language proficiency levels. Specifically, we utilize the capabilities of GPT-3 to generate text modifications based on language proficiency characteristics introduced by the internationally recognized Common European Framework of Reference (CEFR). Based on the application of the widely-used COSMIC model, our robustness analysis discloses that the ERC system’s performance decreased as language proficiency diminished. Hence, this study demonstrates the presence and implications of language proficiency bias in ERC systems, resulting in discriminatory consequences for non-native English speakers. Overall, our perturbation exhibits versatility for diverse analysis objectives. For instance, it allows investigating gender bias and examining unilateral linguistic bias involving native and non-native speakers. By making our implementation publicly accessible, we aim to foster the advancement of fair ERC systems.
INTELLINEO – An Intelligent Personal Assistant

Man-made consciousness means to lessen our physical association with electronic devices. Using voice, body language, and other techniques, this technology is currently being developed to enhance our natural interactions with gadgets. Because they enable us to carry out our day-to-day responsibilities with straightforward commands, virtual personal assistants (VPAs) are rapidly gaining popularity. Virtual personal assistants (VPAs) based on various use cases have been developed by numerous multinational corporations, including intelligent devices like Echodot and Alexa. These virtual assistants are having a drawback of security and data theft, So here IntelliNeo, The intelligent personal assistant is designed in such a way that it addresses the security aspects and also allow us to customize our Virtual Personal Assistant (VPA) based on our needs. The methodology of building IntelliNeo involves Voice Acknowledgment, Text Processing, and the added advantage of active background noise cancellation makes our assistant stand out from the competition. Ultimately the purpose of IntelliNeo is to effortlessly automate the daily tasks of its user. Index Terms pyttsx3, PyQt5 tools, Mail Automation, WhatsApp Automation, Text processing, pyautogui, Google Speech Recognition, Microsoft Sapi5.
Role of Artificial Intelligence in Criminal Investigation

With the advent of technology, criminals are developing more sophisticated methods for criminal activities and the use of Artificial intelligence (AI) has the capacity to address this phenomenon. The role of AI in improving the effectiveness of criminal investigation is discussed in the present study as intelligent scrutiny of crime allows the law enforcement bodies to understand the vast dynamics of criminal activities and it provides potential answers about why how and when about occurrence of crime. Four such technologies-crime prediction, facial recognition, video investigation and neuroimaging are very much into use by criminal investigation bodies. In the current study the use of AI with these technologies in criminal investigation is focused upon. At last, there is always some scope for improvement and the same applies to the use of AI, regarding this the discussion and suggestions are mentioned in this study so that artificial intelligence can be better applied to criminal investigation methods and as this field is in a state of development further research and studies are recommended to derive the maximum benefits from technology.
Distilling from Similar Tasks for Transfer Learning on a Budget

We address the challenge of getting efficient yet accurate recognition systems with limited labels. While recognition models improve with model size and amount of data, many specialized applications of computer vision have severe resource constraints both during training and inference. Transfer learning is an effective solution for training with few labels, however often at the expense of a computationally costly fine-tuning of large base models. We propose to mitigate this unpleasant trade-off between compute and accuracy via semi-supervised cross-domain distillation from a set of diverse source models. Initially, we show how to use task similarity metrics to select a single suitable source model to distill from, and that a good selection process is imperative for good downstream performance of a target model. We dub this approach DistillNearest. Though effective, DistillNearest assumes a single source model matches the target task, which is not always the case. To alleviate this, we propose a weighted multi-source distillation method to distill multiple source models trained on different domains weighted by their relevance for the target task into a single efficient model (named DistillWeighted). Our methods need no access to source data and merely need features and pseudo-labels of the source models. When the goal is accurate recognition under computational constraints, both DistillNearest and DistillWeighted approaches outperform both transfer learning from strong ImageNet initializations as well as state-of-the-art semisupervised techniques such as FixMatch. Averaged over 8 diverse target tasks our multi-source method outperforms the baselines by 5.6%-points and 4.5%-points, respectively. Code: github.com/Kennethborup/DistillWeighted
Sempart: Self-supervised Multi-resolution Partitioning of Image Semantics

Accurately determining salient regions of an image is challenging when labeled data is scarce. DINO-based self-supervised approaches have recently leveraged meaningful image semantics captured by patch-wise features for locating foreground objects. Recent methods have also incorporated intuitive priors and demonstrated value in unsupervised methods for object partitioning. In this paper, we propose Sempart, which jointly infers coarse and fine bi-partitions over an image’s DINO-based semantic graph. Furthermore, Sempart preserves fine boundary details using graph-driven regularization and successfully distills the coarse mask semantics into the fine mask. Our salient object detection and single object localization findings suggest that Sempart produces high-quality masks rapidly without additional post-processing and benefits from co-optimizing the coarse and fine branches.
Quantum Secure Multiparty Summation Based on Quantum Walks

We propose an efficient Quantum Secure Multiparty Summation protocol based on quantum walk states. It enables multiple participants to find the sum of all the data without revealing it to anyone else. With the help of quantum walks, we can achieve this with fewer quantum resources and minimal communication costs. The protocol also requires a semi-honest third party (TP) to facilitate the protocol. The semi-honest TP follows the protocol but attacks passively. Finally, we prove its correctness and security, making it a viable solution for quantum secure multiparty summation.