OpenRaiser/Intern-Atlas huggingface dataset:
Fairness Vs. Personalization: Towards Equity in Epistemic Utility
Venue: arXiv cs.IR 2023
Year: 2023

The applications of personalized recommender systems are rapidly expanding: encompassing social media, online shopping, search engine results, and more. These systems offer a more efficient way to navigate the vast array of items available. However, alongside this growth, there has been increased recognition of the potential for algorithmic systems to exhibit and perpetuate biases, risking unfairness in personalized domains. In this work, we explicate the inherent tension between personalization and conventional implementations of fairness. As an alternative, we propose equity to achieve fairness in the context of epistemic utility. We provide a mapping between goals and practical implementations and detail policy recommendations across key stakeholders to forge a path towards achieving fairness in personalized systems.
Matching Table Metadata with Business Glossaries Using Large Language Models
Venue: arXiv cs.IR 2023
Year: 2023

Enterprises often own large collections of structured data in the form of large databases or an enterprise data lake. Such data collections come with limited metadata and strict access policies that could limit access to the data contents and, therefore, limit the application of classic retrieval and analysis solutions. As a result, there is a need for solutions that can effectively utilize the available metadata. In this paper, we study the problem of matching table metadata to a business glossary containing data labels and descriptions. The resulting matching enables the use of an available or curated business glossary for retrieval and analysis without or before requesting access to the data contents. One solution to this problem is to use manually-defined rules or similarity measures on column names and glossary descriptions (or their vector embeddings) to find the closest match. However, such approaches need to be tuned through manual labeling and cannot handle many business glossaries that contain a combination of simple as well as complex and long descriptions. In this work, we leverage the power of large language models (LLMs) to design generic matching methods that do not require manual tuning and can identify complex relations between column names and glossaries. We propose methods that utilize LLMs in two ways: a) by generating additional context for column names that can aid with matching b) by using LLMs to directly infer if there is a relation between column names and glossary descriptions. Our preliminary experimental results show the effectiveness of our proposed methods.
AdBooster: Personalized Ad Creative Generation using Stable Diffusion Outpainting
Venue: arXiv cs.IR 2023
Year: 2023

In digital advertising, the selection of the optimal item (recommendation) and its best creative presentation (creative optimization) have traditionally been considered separate disciplines. However, both contribute significantly to user satisfaction, underpinning our assumption that it relies on both an item's relevance and its presentation, particularly in the case of visual creatives. In response, we introduce the task of {\itshape Generative Creative Optimization (GCO)}, which proposes the use of generative models for creative generation that incorporate user interests, and {\itshape AdBooster}, a model for personalized ad creatives based on the Stable Diffusion outpainting architecture. This model uniquely incorporates user interests both during fine-tuning and at generation time. To further improve AdBooster's performance, we also introduce an automated data augmentation pipeline. Through our experiments on simulated data, we validate AdBooster's effectiveness in generating more relevant creatives than default product images, showing its potential of enhancing user engagement.
Architecture Knowledge Representation and Communication Industry Survey
Venue: arXiv cs.SE 2023
Year: 2023

Background: The literature offers various methods for capturing software architectural knowledge (AK), including views, viewpoints, and architecture decision records (ADRs). In parallel, sustainability has gained prominence in software engineering, especially concerning software architecture. Nevertheless, practical industry reviews on these subjects seem to be lacking. Aim: In this research we aim to understand the current practice in architecture knowledge, and to explore where sustainability can be applied to address sustainability in software architecture in the future. Method: We used a survey, which utilized a questionnaire containing 34 questions and collected responses from 45 architects working at a prominent bank in the Netherlands, aimed to evaluate the practical representation and communication of architectural knowledge and sustainability. Result: Our analysis yielded two primary discoveries and several intriguing detailed results regarding how AK is captured and conveyed to diverse stakeholders. Firstly, it seems crucial to develop a new architectural element that connects various architectural features and perspectives tailored for different stakeholders. Secondly, providing clear guidance, references, and goals is essential to motivate architects to adopt Sustainable Software Engineering practices. Conclusion: After analysing the data collected through this survey, we have concluded that: a) There are no established domain-specific AK methods/tools in the financial domain. Most practitioners use domain-generic tools. b) A new architectural element that links the various architectural features and viewpoints created for various stakeholders appears to be necessary. c) There is sufficient sustainability awareness and motivation among software architects. However, what they lack are clear guidance, references, and goals to practice sustainable software engineering.
Dataset Factory: A Toolchain For Generative Computer Vision Datasets
Venue: arXiv cs.AI 2023
Year: 2023

Generative AI workflows heavily rely on data-centric tasks - such as filtering samples by annotation fields, vector distances, or scores produced by custom classifiers. At the same time, computer vision datasets are quickly approaching petabyte volumes, rendering data wrangling difficult. In addition, the iterative nature of data preparation necessitates robust dataset sharing and versioning mechanisms, both of which are hard to implement ad-hoc. To solve these challenges, we propose a "dataset factory" approach that separates the storage and processing of samples from metadata and enables data-centric operations at scale for machine learning teams and individual researchers.
Generative AI in Mafia-like Game Simulation
Venue: arXiv cs.AI 2023
Year: 2023

In this research, we explore the efficacy and potential of Generative AI models, specifically focusing on their application in role-playing simulations exemplified through Spyfall, a renowned mafia-style game. By leveraging GPT-4's advanced capabilities, the study aimed to showcase the model's potential in understanding, decision-making, and interaction during game scenarios. Comparative analyses between GPT-4 and its predecessor, GPT-3.5-turbo, demonstrated GPT-4's enhanced adaptability to the game environment, with significant improvements in posing relevant questions and forming human-like responses. However, challenges such as the model;s limitations in bluffing and predicting opponent moves emerged. Reflections on game development, financial constraints, and non-verbal limitations of the study were also discussed. The findings suggest that while GPT-4 exhibits promising advancements over earlier models, there remains potential for further development, especially in instilling more human-like attributes in AI.
Like My Aunt Dorothy: Effects of Conversational Styles on Perceptions, Acceptance and Metaphorical Descriptions of Voice Assistants during Later Adulthood
Venue: arXiv cs.HC 2023
Year: 2023

Little research has investigated the design of conversational styles of voice assistants (VA) for adults in their later adulthood with varying personalities. In this Wizard of Oz experiment, 34 middle-aged (50 to 64 years old) and 24 older adults (65 to 80 years old) participated in a user study at a simulated home, interacting with a VA using either formal or informal language. Older adults with higher agreeableness perceived VA as being more likeable than middle-aged adults. Middle-aged adults showed similar technology acceptance toward the informal and formal VA, and older adults preferred using informal VA, especially those with low agreeableness. Further, while both middle-aged and older adults frequently anthropomorphized VAs by using human metaphors for them, older adults compared formal VA with professionals (e.g., librarians, teachers) and informal VA with their close ones (e.g., spouses, relatives). Overall, the conversational style showed differential effects on the perceptions of middle-aged and older adults, suggesting personalized design implications.
Dr. FERMI: A Stochastic Distributionally Robust Fair Empirical Risk Minimization Framework
Venue: arXiv cs.LG 2023
Year: 2023

While training fair machine learning models has been studied extensively in recent years, most developed methods rely on the assumption that the training and test data have similar distributions. In the presence of distribution shifts, fair models may behave unfairly on test data. There have been some developments for fair learning robust to distribution shifts to address this shortcoming. However, most proposed solutions are based on the assumption of having access to the causal graph describing the interaction of different features. Moreover, existing algorithms require full access to data and cannot be used when small batches are used (stochastic/batch implementation). This paper proposes the first stochastic distributionally robust fairness framework with convergence guarantees that do not require knowledge of the causal graph. More specifically, we formulate the fair inference in the presence of the distribution shift as a distributionally robust optimization problem under $L_p$ norm uncertainty sets with respect to the Exponential Renyi Mutual Information (ERMI) as the measure of fairness violation. We then discuss how the proposed method can be implemented in a stochastic fashion. We have evaluated the presented framework's performance and efficiency through extensive experiments on real datasets consisting of distribution shifts.
ContextRef: Evaluating Referenceless Metrics For Image Description Generation
Venue: arXiv cs.CL 2023
Year: 2023

Referenceless metrics (e.g., CLIPScore) use pretrained vision--language models to assess image descriptions directly without costly ground-truth reference texts. Such methods can facilitate rapid progress, but only if they truly align with human preference judgments. In this paper, we introduce ContextRef, a benchmark for assessing referenceless metrics for such alignment. ContextRef has two components: human ratings along a variety of established quality dimensions, and ten diverse robustness checks designed to uncover fundamental weaknesses. A crucial aspect of ContextRef is that images and descriptions are presented in context, reflecting prior work showing that context is important for description quality. Using ContextRef, we assess a variety of pretrained models, scoring functions, and techniques for incorporating context. None of the methods is successful with ContextRef, but we show that careful fine-tuning yields substantial improvements. ContextRef remains a challenging benchmark though, in large part due to the challenge of context dependence.
MarkNerf:Watermarking for Neural Radiance Field
Venue: arXiv cs.CR 2023
Year: 2023

A watermarking algorithm is proposed in this paper to address the copyright protection issue of implicit 3D models. The algorithm involves embedding watermarks into the images in the training set through an embedding network, and subsequently utilizing the NeRF model for 3D modeling. A copyright verifier is employed to generate a backdoor image by providing a secret perspective as input to the neural radiation field. Subsequently, a watermark extractor is devised using the hyperparameterization method of the neural network to extract the embedded watermark image from that perspective. In a black box scenario, if there is a suspicion that the 3D model has been used without authorization, the verifier can extract watermarks from a secret perspective to verify network copyright. Experimental results demonstrate that the proposed algorithm effectively safeguards the copyright of 3D models. Furthermore, the extracted watermarks exhibit favorable visual effects and demonstrate robust resistance against various types of noise attacks.
Dictionary Attack on IMU-based Gait Authentication
Venue: arXiv cs.CR 2023
Year: 2023

We present a novel adversarial model for authentication systems that use gait patterns recorded by the inertial measurement unit (IMU) built into smartphones. The attack idea is inspired by and named after the concept of a dictionary attack on knowledge (PIN or password) based authentication systems. In particular, this work investigates whether it is possible to build a dictionary of IMUGait patterns and use it to launch an attack or find an imitator who can actively reproduce IMUGait patterns that match the target's IMUGait pattern. Nine physically and demographically diverse individuals walked at various levels of four predefined controllable and adaptable gait factors (speed, step length, step width, and thigh-lift), producing 178 unique IMUGait patterns. Each pattern attacked a wide variety of user authentication models. The deeper analysis of error rates (before and after the attack) challenges the belief that authentication systems based on IMUGait patterns are the most difficult to spoof; further research is needed on adversarial models and associated countermeasures.
JobRecoGPT -- Explainable job recommendations using LLMs
Venue: arXiv cs.AI 2023
Year: 2023

In today's rapidly evolving job market, finding the right opportunity can be a daunting challenge. With advancements in the field of AI, computers can now recommend suitable jobs to candidates. However, the task of recommending jobs is not same as recommending movies to viewers. Apart from must-have criteria, like skills and experience, there are many subtle aspects to a job which can decide if it is a good fit or not for a given candidate. Traditional approaches can capture the quantifiable aspects of jobs and candidates, but a substantial portion of the data that is present in unstructured form in the job descriptions and resumes is lost in the process of conversion to structured format. As of late, Large Language Models (LLMs) have taken over the AI field by storm with extraordinary performance in fields where text-based data is available. Inspired by the superior performance of LLMs, we leverage their capability to understand natural language for capturing the information that was previously getting lost during the conversion of unstructured data to structured form. To this end, we compare performance of four different approaches for job recommendations namely, (i) Content based deterministic, (ii) LLM guided, (iii) LLM unguided, and (iv) Hybrid. In this study, we present advantages and limitations of each method and evaluate their performance in terms of time requirements.
TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification
Venue: arXiv cs.SD 2023
Year: 2023

Audiovisual data is everywhere in this digital age, which raises higher requirements for the deep learning models developed on them. To well handle the information of the multi-modal data is the key to a better audiovisual modal. We observe that these audiovisual data naturally have temporal attributes, such as the time information for each frame in the video. More concretely, such data is inherently multi-modal according to both audio and visual cues, which proceed in a strict chronological order. It indicates that temporal information is important in multi-modal acoustic event modeling for both intra- and inter-modal. However, existing methods deal with each modal feature independently and simply fuse them together, which neglects the mining of temporal relation and thus leads to sub-optimal performance. With this motivation, we propose a Temporal Multi-modal graph learning method for Acoustic event Classification, called TMac, by modeling such temporal information via graph learning techniques. In particular, we construct a temporal graph for each acoustic event, dividing its audio data and video data into multiple segments. Each segment can be considered as a node, and the temporal relationships between nodes can be considered as timestamps on their edges. In this case, we can smoothly capture the dynamic information in intra-modal and inter-modal. Several experiments are conducted to demonstrate TMac outperforms other SOTA models in performance. Our code is available at https://github.com/MGitHubL/TMac.
BalCon -- resource balancing algorithm for VM consolidation
Venue: arXiv cs.DS 2023
Year: 2023

Cloud providers handle substantial number of requests to create and delete virtual machines (VMs) on a daily basis, where the unknown sequence of requests eventually leads to resource fragmentation. To mitigate this issue, periodic consolidation of VMs into fewer number of physical hosts is an important cost-saving procedure, closely related to the vector bin-packing problem. In this paper, we propose the BalCon algorithm for consolidation that steadily reduces the number of active hosts and keeps migration costs low. BalCon classifies the cluster's state and selects one of three heuristics to balance resources for superior consolidation. To evaluate BalCon's performance with respect to optimality, we introduce integer programming models. BalCon finds 99.7% of the optimal solutions for over 750 problem instances. This outstanding result was achieved due to the Force Step of our algorithm, which is the key improvement detail for common heuristics. We compare BalCon with a modified Sercon heuristic using Huawei and synthetic datasets with two resources for allocation.
Latency-Aware Radio Resource Optimization in Learning-Based Cloud-Aided Small Cell Wireless Networks
Venue: arXiv cs.NI 2023
Year: 2023

Low latency communication is one of the fundamental requirements for 5G wireless networks and beyond. In this paper, a novel approach for joint caching, user scheduling and resource allocation is proposed for minimizing the queuing latency in serving user's requests in cloud-aided wireless networks. Due to the slow temporal variations in user requests, a time-scale separation technique is used to decouple the joint caching problem from user scheduling and radio resource allocation problems. To serve the spatio-temporal user requests under storage limitations, a Reinforcement Learning (RL) approach is used to optimize the caching strategy at the small cell base stations by minimizing the content fetching cost. A spectral clustering algorithm is proposed to speed-up the convergence of the RL algorithm for a large content caching problem by clustering contents based on user requests. Meanwhile, a dynamic mechanism is proposed to locally group coupled base stations based on user requests to collaboratively optimize the caching strategies. To further improve the latency in fetching and serving user requests, a dynamic matching algorithm is proposed to schedule users and to allocate users to radio resources based on user requests and queue lengths under probabilistic latency constraints. Simulation results show the proposed approach significantly reduces the average delay from 21% to 90% compared to random caching strategy, random resource allocation and random scheduling baselines.
Simulation-to-reality UAV Fault Diagnosis in windy environments
Venue: arXiv cs.RO 2023
Year: 2023

Monitoring propeller failures is vital to maintain the safe and reliable operation of quadrotor UAVs. The simulation-to-reality UAV fault diagnosis technique offer a secure and economical approach to identify faults in propellers. However, classifiers trained with simulated data perform poorly in real flights due to the wind disturbance in outdoor scenarios. In this work, we propose an uncertainty-based fault classifier (UFC) to address the challenge of sim-to-real UAV fault diagnosis in windy scenarios. It uses the ensemble of difference-based deep convolutional neural networks (EDDCNN) to reduce model variance and bias. Moreover, it employs an uncertainty-based decision framework to filter out uncertain predictions. Experimental results demonstrate that the UFC can achieve 100% fault-diagnosis accuracy with a data usage rate of 33.6% in the windy outdoor scenario.
Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task
Venue: arXiv cs.CL 2023
Year: 2023

We present the joint contribution of Unbabel and Instituto Superior Técnico to the WMT 2023 Shared Task on Quality Estimation (QE). Our team participated on all tasks: sentence- and word-level quality prediction (task 1) and fine-grained error span detection (task 2). For all tasks, we build on the COMETKIWI-22 model (Rei et al., 2022b). Our multilingual approaches are ranked first for all tasks, reaching state-of-the-art performance for quality estimation at word-, span- and sentence-level granularity. Compared to the previous state-of-the-art COMETKIWI-22, we show large improvements in correlation with human judgements (up to 10 Spearman points). Moreover, we surpass the second-best multilingual submission to the shared-task with up to 3.8 absolute points.
A Digital Marketplace Combining WS-Agreement, Service Negotiation Protocols and Heterogeneous Services
Venue: arXiv cs.MA 2023
Year: 2023

With the ever increasing importance of web services and the Cloud as a reliable commodity to provide business value as well as consolidate IT infrastructure, electronic contracts have become very important. WS-Agreement has itself established as a well accepted container format for describing such contracts. However, the semantic interpretation of the terms contained in these contracts, as well as the process of agreeing to contracts when multiple options have to be considered (negotiation), are still pretty much dealt with on a case by case basis. In this paper we address the issues of diverging contracts and varying contract negotiation protocols by introducing the concept of a contract aware marketplace, which abstracts from the heterogeneous offers of different services providers. This allows for the automated consumption of services solely based on preferences, instead of additional restrictions such as understanding of contract terms and/or negotiation protocols. We also contribute an evaluation of several existing negotiation concepts/protocols. We think that reducing the complexity for automated contract negotiation and thus service consumption is a key for the success of future service and Cloud infrastructures.