Reviewerly/exHarmony huggingface dataset:
CoHOZ

Practical image recognition often encounters samples whose labels either are totally unknown or belong to new classes outside the training set. The first problem refers to the open-set recognition (OSR), in which unknown classes are recognized as one with no more semantic information. While the latter is called zero-shot learning (ZSL), in which new classes are usually predefined. The existing literature mostly addresses these two problems separately. In this paper, we take the ambition for solving the combination of these two problems to fulfill semantically recognizing the unknown classes detected in OSR by zero-shot prediction. We propose the Contrastive multimodal prompt tuning for Hierarchical Open-set Zero-shot recognition (CoHOZ). Specifically, we firstly build a global and compatible hierarchical label tree with all downstream datasets aligned, which lays foundations for other modules. To detect unknown classes, we propose the contrastive continuous prompt tuning, which introduces additional negative classes from the fine level of the built hierarchy for prompt learning. To generate candidate classes for zero-shot prediction on the unknown data using prompt, we combine the built hierarchy to collect candidate classes from coarse to fine. In our experiments, when following the standard OSR protocol regarding all the unknown classes as a single class, CoHOZ achieves a new state-of-the-art performance both in unknown detection and open-set recognition. Few-shot tuning by the CoHOZ also shows competitive performance on them. In addition, the detailed semantic information of unknown classes are well explored, which has also been verified in experiments.
Delegate-based Utility Preserving Synthesis for Pedestrian Image Anonymization

The rapidly growing application of pedestrian images has aroused wide concern on visual privacy protection because personal information is under the risk of privacy disclosure. Anonymization is regarded as an effective solution by identity obfuscation. Most recent methods focus on face, but it is not enough when the presence of human body carries lots of identifiable information. This paper presents a new delegate-based utility preserving synthesis (DUPS) approach for pedestrian image anonymization. This is challenging because one may expect that the anonymized image can still be useful in various computer vision tasks. We model DUPS as an adaptive translation process from source to target. To provide a comprehensive identity protection, we first perform anonymous delegate sampling based on image-level differential privacy. To synthesize anonymous images, we then introduce an adaptive translation network and optimize it with a multi-task loss function. Our approach is theoretically sound and can generate diverse results by preserving data utility. The experiments on multiple datasets show that DUPS can not only achieve superior anonymization performance against deep pedestrian recognizers, but also can obtain a better tradeoff between privacy protection and utility preservation compared with state-of-the-art methods.
A Detection Technique Using Dual Authentication Stages Framework for Rogue Access Point Identification

Abstract Wi-Fi hotspot or Access point indicates many threats that lead to compromise a user’s identity, personal data, and network integrity. Therefore, this will uncover the users to the adversaries who deploy Rogue Access Point (RAP) by purpose to pilfer sensitive information. Hence, the adversaries can exchange the communication anonymously or mimic it as a client to pose a threat to public safety. The research aims to propose a new approach based on the beacon frame anomaly to detect the threats of RAP in the Wi-Fi hotspot. This research uses dual-pass identification steps to detect the RAP to identify the anomalies between the first and second responses. First, the identification will check the AP legitimacy to uncover the true identity of an AP. Second, the connection type of RAP is necessary to identify once the AP is determined as a rogue. The five (5) features have been selected in this research: the beacon response, OUI Number, address 1-4, retry bit, and IBSS status. The combination of all the features was 100% successfully detected the presence of RAP and identify its connection source.
A cross-modal UX design pedagogy for industrial design

Everyday experience is multi-sensory, and user experience (UX) design aims to extend this to interactions with products, services, and designed worlds. However, tools and pedagogies for UX are overwhelmingly visual, whereas human-rights-based accessibility legislation mandates the inclusion of diverse peoples, including blind and partially sighted individuals. Coupling auditory and haptic UX techniques from human-computer interaction with industrial design’s (ID) cross-modal tradition of prototyping physical products fostered our novel cross-modal UX course for second-year ID undergraduates. Affordance-based theories of perception-action and Gestalt principles of perceptual organization were used to inform design in auditory, tactile, and visual sensory modalities situated in a novel pedagogical framework. Each week theoretical models were presented alongside hands-on workshops using the BBC micro:bit, developing computational literacy through cross-modal physical prototyping. Student projects demonstrate an understanding of theory and practice and include auditory and tactile interfaces.
Performance Improvement of Grid Mapping K-Means with the Average Value at Grid Point

Abstract K-Means is a method that is well-known for its ability to handle large datasets, but is often stuck in a local optima state. This issue happens because K-Means generally uses random numbers to serve as the center point (centroid) of each cluster, and places each instance based on the proximity of the distance using Euclidean Distance. Hence, the concept of density parameter was developed, which tries to determine the ideal centroid based on the determination of several grid points in each existing cluster. The concept is known as Grid Mapping. In simple terms, Grid Mapping K-Means is a method in which each cluster is divided into several grid points, and then the centroid is randomly determined from each existing grid point. Further, the grid point with the largest number of instance is used as the initial centroid of each cluster. However, this method certainly has a weakness, since there is a possibility that the initial centroid on the generated grid point is not the best initial centroid. Therefore, this study was conducted to test the determination of initial centroid by Grid Mapping K-Means.
Analysis of Flow Overview Using Empirical Equations In The River Kampung Bugis Tarakan City

Abstract During the rainy season, rivers in the Tarakan region experience overflows due to high rainfall, household waste and, river silting (sedimentation). On the Kampung Bugis and the Mahakam River in Kampung Empat, they overflow when it rains with intensity and a long duration of time. This study aims to determine the potential for overflow and the factors causing the overflow of the Kampung Bugis and the Mahakam Rivers in Kampung Empat by using the HEC-RAS software. Based on the analysis results of calculations and simulations with HEC-RAS software in the Kampung Bugis river at 10 stations (STA), the maximum discharge was obtained with values (QFlow + Q100th) = 51.44 m3/s and (QCanal) = 41.58 m3/s. From these results, the comparison is that (QFlow + Q100th) &gt; (QCanal) which means that the Canal cannot accommodate the flowing discharge. The overflow factor is due to the sedimentation along the river Canal and the high rainfall.
The reality of the reproduction. Aesthetics of a ”conscious” approach to sound design in the soundscape composition: a case study

In this paper the processes that led to the composition of the piece Impressioni In-Naturali (for digital support and live performer) will be exposed. Special attention is given to the performance practice as a construction of the listening experience in an electroacoustically configured space and as a form of ecological awareness. It was also necessary to define an acousmatic form not fixed but adaptive to the context, underlining an active attitude highlighted mainly in live electronics. First, the theoretical assumptions relating to the genesis of the composition will be illustrated and then an analysis is provided of the choices of selection and structuring of the material at the microscopic level (realization of the synthesis sounds) and macroscopic (structuring of the generated material) and the practice of diffusion in relation to an informatic performance environment.
Harmonic Anamorphism in an Interactive Improvisation

The avant-garde music in the 20th century brought challenges to musicians of several genres by means of rhythmic and technological innovations. In this context, rhythm manuals and exercise books emerged as a propaedeutic, such as the work of José Eduardo Gramani, to help those aspiring to the 'contemporary music'. Despite the successful application of his books in the pedagogical area, Gramani's 'Rhythmic Series', specifically, found little creative use in popular music. Another device that met little use in popular music was Steve Reich's phase-shifting processes. Hence, we questioned: ‘how Gramani's Series and Reich's phase-shifting could be combined and applied in popular piano music?’ We have used live looping to mix rhythmic timelines by a phase-shifting procedure, in consequence, engendered harmonic anamorphism as a base for an improvisational interaction. To highlight part of our results, we present the piece Entreformas, composed for piano improvisation, performed with a hands-free live looping technique using the DAW Reaper.
Time and Memory Efficient Large-Scale Canonical Correlation Analysis in Fourier Domain

Canonical correlation analysis (CCA) is a linear correlation analysis technique used widely in the statistics and machine learning community. However, the high complexity involved in pursuing eigenvector lays a heavy burden on the memory and computational time, making CCA nearly impractical in large-scale cases. In this paper, we attempt to overcome this issue by representing the data in the Fourier domain. Thanks to the data characteristic of pattern repeatability, one can translate projection-seeking of CCA into choosing some discriminative Fourier bases with only element-wise dot product and sum operations, without time-consuming eigenvector computation. Another merit of this scheme is that the eigenvalues can be approximated asymptotically in contrast to existing methods. Specifically, the eigenvalues can be estimated progressively, and the accuracy goes up as the number of data samples increases monotonously. This makes it possible to use partial data samples to obtain satisfactory accuracy. All the facts above make the proposed method extremely fast and memory efficient. Experimental results on several large-scale datasets, such as MNIST 8M, X-RAY MICROBEAM SPEECH, and TWITTER USERS Data, demonstrate the superiority of the proposed algorithm over SOTA large-scale CCA methods, as our proposed method achieves almost same accuracy with the training time being 1,000 times faster than SOTA methods.
The iOSCahedron: developing a hybrid Spatialization Instrument

Real time sound spatialization has gained more and more attention during the last decades inside and outside of academic contexts. In parallel, a number of new instruments (NIMEs) have been developed that consider spatialization as a key element for musical performances. While an increasing interest in intelligent instruments that explore machine agency and autonomous behavior can be observed, instruments for sound spatialization that refer to similar concepts are still limited. By mapping the historical conditions of contemporary sound technologies and their colonial ancestors a methodological thread for the development of a spatialization instrument will be suggested. Based on that a hybrid instrument - The iOSCahedron - for both sound processing and spatialization in real time will be presented. Due to its design, it is a dynamically and organically responding Digital Musical Instrument in the shape of an icosahedron1 constructed as tensegrity2 that transmits motion and stretch sensor data via OSC (Open Sound Control)3 .
Pay Attention to Your Positive Pairs: Positive Pair Aware Contrastive Knowledge Distillation

Deep neural networks have achieved impressive success on various multimedia applications in the past decades. To reach a higher performance on real-world resource-constrained devices with large models that are already learned, knowledge distillation, which aims at transferring representational knowledge from a large teacher network into a small student network, has attracted increasing attention. Recently, contrastive distillation methods have achieved superior performance in this area, due to the powerful representability brought by contrastive/self-supervised learning. These models often transfer knowledge through individual samples or inter-class relationships, while ignoring the correlation lying among intra-class samples, which convey abundant information. In this paper, we propose a Positive pair Aware Contrastive Knowledge Distillation (PACKD) framework to extend the contrastive distillation with more positive pairs to capture more abundant knowledge from the teacher. Specifically, it pulls together features of pairs from the same class learned by the student and teacher while simultaneously pushing apart those of pairs from different classes. With a positive-pair similarity weighting strategy based on optimal transport, the proposed contrastive objective is able to improve the feature discriminability between positive samples with large visual discrepancies. Experiments on different benchmarks demonstrate the effectiveness of the proposed PACKD.
Knowledge Guided Representation Disentanglement for Face Recognition from Low Illumination Images

Low illumination face recognition is challenging as details are lacking due to lighting conditions. Retinex theory points out that images can be divided into reflectance with color constancy and ambient illumination. Inspired by this, we propose a knowledge-guided representation disentanglement method to disentangle facial images into face-related and illumination-related features, and then leverage the disentangled face-related features for face recognition. Specifically, the proposed method consists of two components: feature disentanglement and face classifier. Following Retinex, high-dimensional face-related features and ambient illumination-related features are extracted from facial images. Reconstruction and crossreconstruction methods are used to make sure the integrity and accuracy of the disentangled features. Furthermore, we find that the influence of illumination changes on illumination-related features should be invariant for faces of different identities, so we design an illumination offset loss to satisfy the prior invariance for better disentanglement. Finally high-dimensional face-related features are mapped to low-dimensional features through the face classifier for use in face recognition task. Experimental results on low illumination and NIR-VIS datasets demonstrate the superiority and effectiveness of our proposed method.
Generating Smooth and Facial-Details-Enhanced Talking Head Video

Talking head video generation has received increasing attention recently. So far the quality (especially the facial details) of the videos output from state-of-the-art deep learning methods is limited by either the quality of training data or the performance of generators, and needs to be further improved. In this paper, we propose a data pre- and post- processing strategy based on a key observation: generating talking head video from multi-modal input is a challenging problem and generating smooth video with fine facial details makes the problem even harder. Then we propose to decompose the problem solution into a main deep model, a pre- and a post- processing. The main deep model generates a reasonably good talking face video, with the aid of a pre-process, which also contributes to a post-process for restoring smooth and fine facial details in the final video. In particular, our main deep model reconstructs a 3D face from an input reference frame, and then uses an AudioNet to generate a sequence of facial expression coefficients with an input audio clip. To ensure final facial details in the generated video, we sample the original texture from the reference frame in the pre-process with the aid of reconstructed 3D face and a predefined UV map. Accordingly, in the post-process, we smooth the expression coefficients of adjacent frames to alleviate jitters and apply a pretrained face restoration module to recover the fine facial details. Experimental results and ablation study show the advantage of our proposed method.
Situational Perception Guided Image Matting

Most automatic matting methods try to separate the salient foreground from the background. However, the insufficient quantity and subjective bias of the current existing matting datasets make it difficult to fully explore the semantic association between object-to-object and object-to-environment in a given image. In this paper, we propose a Situational Perception Guided Image Matting (SPG-IM) method that mitigates subjective bias of matting annotations and captures sufficient situational perception information for better global saliency distilled from the visual-to-textual task. SPG-IM can better associate inter-objects and object-to-environment saliency, and compensate the subjective nature of image matting and its expensive annotation. We also introduce a textual Semantic Transformation (TST) module that can effectively transform and integrate the semantic feature stream to guide the visual representations. In addition, an Adaptive Focal Transformation (AFT) Refinement Network is proposed to adaptively switch multi-scale receptive fields and focal points to enhance both global and local details. Extensive experiments demonstrate the effectiveness of situational perception guidance from the visual-to-textual tasks on image matting, and our model outperforms the state-of-the-art methods. We also analyze the significance of different components in our model.
CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training

With the flourishing of social media platforms, vision-language pre-training (VLP) recently has received great attention and many remarkable progresses have been achieved. The success of VLP largely benefits from the information complementation and enhancement between different modalities. However, most of recent studies focus on cross-modal contrastive learning (CMCL) to promote image-text alignment by pulling embeddings of positive sample pairs together while pushing those of negative pairs apart, which ignores the natural asymmetry property between different modalities and requires large-scale image-text corpus to achieve arduous progress. To mitigate this predicament, we propose CMAL, a <u>C</u>ross-<u>M</u>odal <u>A</u>ssociative <u>L</u>earning framework with anchor points detection and cross-modal associative learning for VLP. Specifically, we first respectively embed visual objects and textual tokens into separate hypersphere spaces to learn intra-modal hidden features, and then design a cross-modal associative prompt layer to perform anchor point masking and swap feature filling for constructing a hybrid cross-modal associative prompt. Afterwards, we exploit a unified semantic encoder to learn their cross-modal interactive features for context adaptation. Finally, we design an associative mapping classification layer to learn potential associative mappings between modalities at anchor points, within which we develop a fresh self-supervised associative mapping classification task to boost CMAL's performance. Experimental results verify the effectiveness of CMAL, showing that it achieves competitive performance against previous CMCL-based methods on four common downstream vision-and-language tasks, with significantly fewer corpus. Noteably, CMAL obtains new state-of-the-art results on SNLI-VE and REC (testA).
When True Becomes False

Recently, the link prediction task on Hyper-relational Knowledge Graphs (HKGs) has been a hot spot, which aims to predict new facts beyond binary relations. Although previous models have accomplished considerable achievements, there remain three challenges: i) the previous models neglect the existence of False Positive Entities (FPEs), which are true entities in the binary triples, yet becomes false when encountering the query statements of HKGs; ii) Due to the sparse interactions, the models are not capable of coping with long-tail hyper-relations, which are ubiquitous in the real-world; iii) The models are generally transductive learning processes, and have difficulty in adapting new hyper-relations. To tackle the above issues, we firstly propose the task of few-shot link prediction on HKGs and devise hyper-relation-aware attention networks with a contrastive loss, which are empowered to encode all entities including FPEs effectively and increase the distance between the true entities and FPEs through contrastive learning. With few-shot references available, the proposed model then learns the representations of their long-tail hyper-relations and predicts new links by calculating the likelihood between queries and references. Furthermore, our model is inductive and can be scalable to any new hyper-relation effortlessly. Since it is the first trial on few-shot link prediction for HKGs, we also modify the existing few-shot learning approaches on binary relational data to work with HKGs as baselines. Experimental results on three real-world datasets show the superiority of our model over various state-of-the-art baselines.
Clustering Generative Adversarial Networks for Story Visualization

Story visualization aims to generate a series of images, semantically matching a given sequence of sentences, one for each, and different output images within a story should be consistent with each other. Current methods generate story images by using a heavy architecture with two generative adversarial networks (GANs), one for image quality, and one for story consistency, and also rely on additional segmentation masks or auxiliary captioning networks. In this paper, we aim to build a concise and single-GAN-based network, neither depending on additional semantic information nor captioning networks. To achieve this, we propose a contrastive-learning- and clustering-learning-based approach for story visualization. Our network utilizes contrastive losses between language and visual information to maximize the mutual information between them, and further extends it with clustering learning in the training process to capture semantic similarity across modalities. So, the discriminator in our approach provides comprehensive feedback to the generator, regarding both image quality and story consistency at the same time, allowing to have a single-GAN-based network to produce high-quality synthetic results. Extensive experiments on two datasets demonstrate that our single-GAN-based network has a smaller number of total parameters in the network, but achieves a major step up from previous methods, which improves FID from 78.64 to 39.17, and FSD from 94.53 to 41.18 on Pororo-SV, and establishes a strong benchmark FID of 76.51 and FSD of 19.74 on Abstract Scenes.