<figure><img src="image_1.png" /></figure>

HAL
open science

# CoPubli: a minimal software pipeline to extract, browse and geolocalise HAL coauthors

Kumar Guha, Maria Kazolea, Luigi Liquori, Andrea Nebot, Daniel da Silva

► To cite this version:

Kumar Guha, Maria Kazolea, Luigi Liquori, Andrea Nebot, Daniel da Silva. CoPubli: a minimal software pipeline to extract, browse and geolocalise HAL coauthors. 2026. hal-05436705v2

HAL Id: hal-05436705

https://inria.hal.science/hal-05436705v2

Preprint submitted on 15 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# CoPubli: a minimal software pipeline
to extract browse and geolocalise HAL co-authors

Kumar Guha (1), Maria Kazolea (2), Luigi Liquori (3),

Andrea Nebot (1), Daniel Da Silva (1)

1 INRIA-IES - Information et Édition Scientifiques, France

2 CARDAMOM, Inria, France

3 KAIROS, Inria, France

## Abstract

This publication describes CoPubli, a software created to retrieve Inria publications involving foreign co-authors from the HAL open archive, in order to help identify and develop international cooperation between Inria and research institutions from around the world. The software uses the XML-TEI rendering of the HAL open archive API and analyses, with Python Jupyter Notebook, the contents to determine the city of each stated institution. A Spreadsheet and a Web Dashboard offer different views of the results, such as a map, a keyword cloud, four year evolution of copublications. The views can be refined combining different filters (Research team, foreign institution, city, country, etc.)

**Keywords:** datascience, dashboard, international scholarly publications, HAL, open archive, open science

## Introduction

We aimed to solve the following query : "*For each Inria Centre (or Inria Team, or Inria Researcher): compile a list of foreign co-publishing researchers along with their research institutions, city, and country between 2018 and 2024*"

For Inria, the vast majority of the Institute's publications are available on the national open archive platform. HAL<fnref n="1" /> (short for *Hyper Articles en Ligne* 'Hyper Articles Online') is an open archive where authors can deposit scholarly documents from all academic fields.

Indeed, their inclusion in HAL is a prerequisite for feeding the activity reports of the research teams. Furthermore, the mandatory deposit of full texts, implemented in 2015—a year before the enactment of the "*Loi pour une république numérique*<fnref n="2" />" (Digital Republic Act) enables us to access over 90% of the full texts of the recorded publications.

Inria's publications are also referenced in aggregators, publisher platforms, and open science databases such as [OpenAlex](https://openalex.org/). However, to date, only HAL provides the granularity of author affiliations specific to Inria that we require. In HAL, Inria authors are primarily affiliated with their Research Team, which is linked to one of Inria's nine centers in France<fnref n="3" /> or to a "component" of a center. It is at the level of teams and centers that Inria analyzes its scientific output to inform its strategic decisions and report on its activities.

Additionally, an Inria team within the Scientific Information and Publishing Service ensures the quality of metadata, particularly regarding Inria and external affiliations, recognizing the Institute's needs in this area.

Thus, HAL logically remains the platform that, to this day, offers the precision and quality of metadata necessary for this study.

HAL also comes with a complete API<fnref n="4" /> allowing to extract metadata. Browsing and extracting the HAL dataset using the HAL API ([AuréHAL](https://auréhal.inria.fr/)) is rather simple: unfortunately, One of the limitations concerns the city. The city can be identified in the address, which is a free-text field in HAL's database. Therefore, a method will be needed to determine the city in order to obtain its latitude and longitude based on different information provided by the metadata. Another limitation of the API rendering is that the country of the authors' affiliation is not stated directly inside the author metadata and has to be retrieved through its PID, making it difficult to immediately identify international copublications. CoPubli solves these issues by a software pipeline.

## 1 Use Cases

We envisage at least three use cases for CoPubli, all depending on how frequently we want to run the pipeline and access the results:

**On demand:** this uses case triggers the pipeline on HAL upon request and makes the spreadsheet file available for download, along with a dashboard web view for browsing.

**Automatic nightly build:** this uses case runs the pipeline on HAL every night and makes the spreadsheet available for download, along with a dashboard web view for browsing.

<footnote n="1">¹ https://hal.science or https://inria.hal.science</footnote>

<footnote n="2">² https://www.legifrance.gouv.fr/loda/id/JORFTEXT000033202746/</footnote>

<footnote n="3">³ https://www.inria.fr/en/inria-ecosystem</footnote>

<footnote n="4">⁴ Documentation API-HAL | API Archive Ouverte HAL (in French)</footnote>
```xml
<listOrg type="structures">
  <org type="institution" xml:id="struct-19908" status="VALID">
    <idno type="IdRef">026461307</idno>
    <idno type="ROR">https://ror.org/01aj84f44</idno>
    <orgName>Aarhus University [Aarhus]</orgName>
    <desc>
      <address>
        <addrLine>Nordre Ringgade 1 DK-8000 Aarhus C</addrLine>
        <country key="DK"/>
      </address>
      <ref type="url">http://www.au.dk/en/</ref>
    </desc>
  </org>
</listOrg>
```

## 2.2 Using the API and analysing the metadata

The first version of the Pipeline analyses each record given by the API and keeps metadata when a non-French country if found. However, this process takes a long time and an improvement is to extract all the records first (by 1000 records batch), and then to use the extracted files to create the dataset.

Using the XML-TEI records from HAL, the pipeline - a Jupyter Notebook script - creates three dataframes: foreign institutions, French institutions and authors. Each dataframe includes the AuréHal identifier of the institution, along with additional relevant information for visualization purposes.

EU/Non-EU Distinction: using the list of country codes for European Union member states, we classify each country as either an EU or non-EU member.

To enhance readability, country codes (ISO 3166-1 alpha-2) are converted into full country names (e.g., DK→Denmark) using the restcountries API.

Additionally, the acronyms of Inria's project teams are included, serving as a filtering criterion for result consultation.

The list of authors is structured so that each entry corresponds to a single author affiliation. Consequently, if an Inria author has multiple affiliations, only the entry corresponding to their affiliation with the project team is retained. We also isolate authors with only non-French affiliations. All other authors are deleted.

## 2.3 Spreadsheet extraction using HAL API

The first part of the pipeline produces a dataset that is converted into an Spreadsheet file that can be used for analysis. Solving the query means identifying each Inria author in each publication, and each foreign co-author, as well as their institution and country.

Each record in the spreadsheet file contains the following information: Inria Team, Inria Research Centre, Full name of the French author, Full name of the co-author(s), Co-author Institution, Address, Country, AUREHAL_ID, EU (flag), Year, HallID, Domain(s), Keywords, Abstract.

The output is reproducible for all of Inria's scientific output and can be applied to other time periods.

<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th><th>E</th><th>F</th><th>G</th><th>H</th><th>I</th><th>J</th><th>K</th><th>L</th></tr><tr><th></th><th>Centre</th><th>Equipe</th><th>Auteurs FR</th><th>Auteurs copubliants</th><th>Organisme copubliant</th><th>Adresse</th><th>Pays</th><th>ID Aurehal</th><th>Non U</th><th>Année</th><th>HallID</th><th>Domaine(s)</th></tr></thead><tbody><tr><td>2</td><td>Paris</td><td>ALMAnaCH</td><td>Romary, Laure</td><td>Adli, Aria</td><td>Universität zu Köln = University of Col</td><td>Albertus-Magnus-Platz, 50923 Köln</td><td>Germany</td><td>463785</td><td>UE</td><td>2017</td><td>hal-01876327</td><td>Computer Science</td></tr><tr><td>3</td><td>Paris</td><td>ALMAnaCH</td><td>Romary, Laure</td><td>Engel, Eric</td><td>Universität zu Köln = University of Col</td><td>Albertus-Magnus-Platz, 50923 Köln</td><td>Germany</td><td>463785</td><td>UE</td><td>2017</td><td>hal-01876327</td><td>Computer Science</td></tr><tr><td>4</td><td>Paris</td><td>ALMAnaCH</td><td>Romary, Laure</td><td>Same, Fahime</td><td>Universität zu Köln = University of Col</td><td>Albertus-Magnus-Platz, 50923 Köln</td><td>Germany</td><td>463785</td><td>UE</td><td>2017</td><td>hal-01876327</td><td>Computer Science</td></tr><tr><td>5</td><td>Saclay</td><td>PARIETAL</td><td>Ciuciu, Philipp</td><td>Kolenic, Marian</td><td>National Institute of Mental Health</td><td>T</td><td>National Institute of Mental Health</td><td>Czechia</td><td>553940</td><td>UE</td><td>2018</td><td>cea-01883283</td><td>Statistics [stat]/M<sup>a</sup></td></tr><tr><td>6</td><td>Saclay</td><td>PARIETAL</td><td>Ciuciu, Philipp</td><td>Löfstedt, Tommy</td><td>Umeå University = Umeå Universitet</td><td>Umeå University901 87 Umeå, Sweden</td><td></td><td>548056</td><td>UE</td><td>2018</td><td>cea-01883283</td><td>Statistics [stat]/M<sup>a</sup></td></tr><tr><td>7</td><td>Saclay</td><td>PARIETAL</td><td>Ciuciu, Philipp</td><td>Spaniel, Filip</td><td>National Institute of Mental Health</td><td>T</td><td>National Institute of Mental Health</td><td>Czechia</td><td>553940</td><td>UE</td><td>2018</td><td>cea-01883283</td><td>Statistics [stat]/M<sup>a</sup></td></tr><tr><td>8</td><td>Saclay</td><td>PARIETAL</td><td>Jardri, Renaud</td><td>Kolenic, Marian</td><td>National Institute of Mental Health</td><td>T</td><td>National Institute of Mental Health</td><td>Czechia</td><td>553940</td><td>UE</td><td>2018</td><td>cea-01883283</td><td>Statistics [stat]/M<sup>a</sup></td></tr><tr><td>9</td><td>Saclay</td><td>PARIETAL</td><td>Jardri, Renaud</td><td>Löfstedt, Tommy</td><td>Umeå University = Umeå Universitet</td><td>Umeå University901 87 Umeå, Sweden</td><td></td><td>548056</td><td>UE</td><td>2018</td><td>cea-01883283</td><td>Statistics [stat]/M<sup>a</sup></td></tr><tr><td>10</td><td>Saclay</td><td>PARIETAL</td><td>Jardri, Renaud</td><td>Spaniel, Filip</td><td>National Institute of Mental Health</td><td>T</td><td>National Institute of Mental Health</td><td>Czechia</td><td>553940</td><td>UE</td><td>2018</td><td>cea-01883283</td><td>Statistics [stat]/M<sup>a</sup></td></tr><tr><td>11</td><td>Saclay</td><td>PARIETAL</td><td>Jardri, Renaud</td><td>de Pierrefeu, Amici</td><td>Umeå University = Umeå Universitet</td><td>Umeå University901 87 Umeå, Sweden</td><td></td><td>548056</td><td>UE</td><td>2018</td><td>cea-01883311</td><td>Statistics [stat]/M<sup>a</sup></td></tr><tr><td>12</td><td>Saclay</td><td>PARSIFAL</td><td>Miller, David</td><td>Arndt, Thomas</td><td>Evonik Industries AG [Darmstadt]</td><td>Kirschenallee, 64293 Darmstadt</td><td>Germany</td><td>1075038</td><td>UE</td><td>2018</td><td>cea-02912335</td><td>Engineering Scienc</td></tr><tr><td>13</td><td>Saclay</td><td>PARSIFAL</td><td>Miller, David</td><td>Askins, Steve</td><td>Universidad Politécnica de Madrid</td><td>Calle Ramiro de Maeztu, 7, 28040 Spain</td><td></td><td>302798</td><td>UE</td><td>2018</td><td>cea-02912335</td><td>Engineering Scienc</td></tr><tr><td>14</td><td>Saclay</td><td>PARSIFAL</td><td>Miller, David</td><td>Bengoechea, Jaione</td><td>Centro Nacional de Energias Renovab</td><td>Sarriguren (Navarra) - España</td><td>Spain</td><td>347030</td><td>UE</td><td>2018</td><td>cea-02912335</td><td>Engineering Scienc</td></tr><tr><td>15</td><td>Saclay</td><td>PARSIFAL</td><td>Miller, David</td><td>Carter, Sam</td><td>RayGen Resources Pty Ltd [Blackburn]</td><td>15 King Street, Blackburn, VIC 3130</td><td>Australia</td><td>1075043</td><td>Non UE</td><td>2018</td><td>cea-02912335</td><td>Engineering Scienc</td></tr></tbody></table>

## 2.4 Enriching Spreadsheet with cities, latitude and longitude

The second part of the software enriches the obtained information with cities, latitudes and longitudes, using Geonames open data<fnref n="7" /> file "cities500.txt", containing a list of cities of at least 500 inhabitants.

<footnote n="7">⁷ https://download.geonames.org/export/dump/</footnote>
Each time a new affiliation is typed into HAL, it is registered in the AuréHal database. The minimum information required for the user is the name and the country and the type of institution (research team, laboratory, etc.). This is clearly not enough to correctly identify an institution.

This is why, unlike many open repositories, HAL is monitored by research librarians who validate, when they can, the new institutions, completing missing information. One common practice is to add the name of the city in brackets after the name of the institution. Our software uses this information, combined with the country name, to retrieving latitude and longitude. Some countries have the same city name in different districts, provinces or states. We found that the United States were the country where the risk of finding a homonym in Geonames was the highest, so we also analyse the addresses in order to retrieve the State code or name.

When the information in brackets is absent, we have to look for a city name in the organisation name or in the address, when it is present, to try and identify the name of a city, based on the Geonames cities500 dataset.

Using this method, we found about 80% of the cities of all the stated institutions with addresses. This allows us to use the latitude and longitude for the Dashboard.

When we have 100% certainty for the city information, we store the result into a dictionary which can be used as reference for the next updates<fnref n="8" />.

Another limitation of the HAL database is that it does not require authors to specify the highest level of hierarchy of foreign institutions. For example, an author may affiliate their publication with the Dipartimento di Matematica (DiMa) without specifying that DiMa is under the supervision of the University of Genova (UniGe), or they may directly affiliate the publication with UniGe. However, as the affiliations are recorded in the Auréhal reference database and this database is monitored, this hierarchy is often retrievable.

We are currently working on an improvement, to take into consideration the highest level of hierarchy of each organisation of the co-authors, which would help determine the publication collaborations at the institutional level (and not the laboratory or research team level) and also add information to retrieve the city.

# 3 Browsing the dataset via a Web Dashboard

The third component of the CoPubli pipeline is an interactive web-based dashboard designed to facilitate exploration and analysis of international co-publications extracted from the enriched Spreadsheet dataset. This dashboard offers a user-friendly interface allowing stakeholders to browse the data both globally and at more granular levels (by Inria Centre, team, country, city, year, or collaborating institution). It enables both descriptive analysis and rapid identification of collaboration patterns through interactive visualisations.

The dashboard is composed of several complementary views, organised into thematic tabs: a main overview, a keyword exploration view (wordcloud), a publication collaboration network view, and an evolution view. These views are dynamically updated through global filters and can be combined to narrow the analysis to specific scopes.

All charts are implemented using interactive plotting libraries that support dynamic filtering and exploration through hover, zoom, and selection tools. The dashboard is conceived as a lightweight front-end layer on top of the enriched Spreadsheet dataset, making it suitable both for on-demand execution and for automated updates (nightly builds).

## 3.1 Dashboard structure and filtering logic

A set of global filters is displayed at the top of the interface, allowing the user to select: Inria centre, Inria team, Country, City, Co-author institution, Publication year.

The filters are cumulative and trigger an update of all charts simultaneously, providing a coherent analytical framework.

This enables users to move seamlessly from an overall landscape to a focused subset of collaborations (for example: one centre, one year, and one partner country).

<footnote n="8">⁸ https://github.com/INRIA/datalake/blob/main/POC%20-%20R%C3%A9seaux%20de%20copublications/ID_Aurehal_Ville_Etat_Latitude_Longitude.xlsx</footnote>
<figure><img src="image_2.png" /></figure>

COPUBLICATIONS INRIA

<figure><img src="image_3.png" /></figure>

Copublications internationales – tous les centres Inria, toutes les équipes, toutes les villes, tous les pays (toutes les années)

<figure><img src="image_4.png" /></figure>

The dashboard is built using interactive components (dropdown menus, dynamic callbacks) that query and filter the dataset in memory. All visualisations are rendered dynamically based on the filtered dataset.

The dashboard has a multi-panel layout combining filter components (dropdown selectors), metric panels (KPI cards), interactive chart containers (map + charts) and tab navigation for modular exploration.

## 3.2 Main overview tab: global indicators and geographical distribution

The main view provides a global synthetic overview of the filtered dataset. It includes:

### (a) KPI cards (summary indicators)

A set of numerical indicators is shown to provide instant context about the selected scope, including: Total number of publications, Number of unique cities, Number of unique countries, Number of Inria teams, Number of Inria authors, Number of co-authors / collaborators, These indicators are computed as aggregations over the filtered dataset (counts of unique values per category).

### (b) World map of international co-publications

A geographical representation displays the spatial distribution of collaborations. Each point corresponds to a city associated with at least one collaborating institution, using the latitude/longitude previously added during the enrichment phase.

The map uses bubble markers whose size reflects the number of publications (or collaboration occurrences) associated with a city, allowing users to detect major hotspots. A colour legend distinguishes countries, enabling comparative interpretation.

The map supports interactive navigation (zoom, drag, hover information). Hovering over a point displays contextual information such as the city name, country, and collaboration volume.

### (c) Publications by Inria centre

A summary distribution is provided showing the number of publications per Inria centre, allowing direct comparison between centres within the selected scope.

## 3.3 Publication trends and Top collaborators

A bar chart displays the number of publications per year in the selected scope. This visual allows identification of: Trends in collaboration growth or decline, Years of peak activity, Stagnation periods.

The chart is computed by grouping the dataset by year and counting publications, and it updates dynamically according to the filters.

Three ranked visualisations (top 10) are presented to highlight dominant collaboration partners:

### (a) Top 10 partner countries

Displayed as a donut chart representing the share of publications associated with each country. This gives a macro-level view of international cooperation distribution.

### (b) Top 10 partner cities

Displayed as a donut chart representing the most frequent collaboration locations. This helps identify urban hubs of scientific cooperation.

### (c) Top 10 collaborating institutions

Displayed as a donut chart listing the most frequent partner institutions. Hover interactions (or legends) provide readability for long institution names.

All top 10 charts are computed by grouping the dataset by the corresponding category and selecting the most frequent values.
<figure><img src="image_5.png" /></figure>

## 3.4 Wordcloud tab: keyword exploration

The keyword wordcloud tab provides an exploratory thematic view of the scientific output. It is built from the keywords declared in HAL metadata. The size of each word is proportional to its frequency in the filtered dataset. This allows rapid identification of dominant research topics for a given centre, country, year, etc.

A limitation is that the wordcloud only reflects the subset of publications containing explicit keywords in HAL (approximately half of the dataset). This information is explicitly indicated in the dashboard.

A further improvement could include abstract and keyword, as well as Inria keywords associated with each research team.

## 3.5 Publication network tab: collaboration graph

A network visualisation represents co-publication relationships between entities such as Inria authors and foreign co-authors (or institutions), depending on the implementation. Nodes represent actors (authors or organisations) and edges represent co-publication links, with weight proportional to collaboration frequency.

This view enables detection of: Major collaboration hubs, Highly connected co-authors, International bridges between teams and countries (the latter can be part of a further development)

Interactive features allow hovering to display metadata (e.g., author name, affiliation, number of collaborations), and zooming/panning to explore dense subgraphs.

Réseau de copublications — Plein écran

Fermer

<figure><img src="image_6.png" /></figure>
## 3.6 Structural trends and collaboration dynamics

The "Evolution of co-publications" tab is designed to highlight how international collaborations evolve over time and how different entities (centres, teams, countries, cities, institutions) interact across publication years. Unlike the main overview tab, which focuses on distribution and ranking, this view provides analytical charts dedicated to temporal dynamics and structural composition of collaborations.

This tab is composed of several complementary visualisations, including a Sankey flow diagram and radar/spider charts, which enable multi-dimensional comparison between entities.

### (a) Sankey diagram: collaboration flows between dimensions

A Sankey diagram is used to represent collaboration flows between several categorical dimensions (e.g., Inria centre → team → partner country → partner institution, or centre → country → city depending on the selected configuration).

Each node corresponds to a category value (e.g., a given centre or country), while each link represents the number of publications associated with the connection between two categories. The thickness of links is proportional to the number of publications, making the diagram particularly effective for identifying:

* main collaboration routes (dominant centre-team),

* concentration or dispersion of partnerships,

* major partner institutions tied to specific centres or teams.

### (b) Radar chart: scientific domain profiles by Inria centre

A radar chart (also called spider plot) is used to represent and compare the scientific domain distribution of publications across Inria centres.

Each axis corresponds to a scientific domain (as defined in the dataset), and the plotted values represent the volume or proportion of publications associated with each domain for a given centre.

This visualisation allows users to identify the dominant domains of each centres.

### (c) Temporal distribution charts: evolution by partner dimensions

In addition to Sankey and radar, the evolution tab may include temporal evolution charts focusing on specific partner dimensions. These graphs represent how collaborations change over time when grouped by Countries, institutions, Inria centres, Inria teams.

These charts may take the form of:

* line charts (publication counts over years),

* stacked bar charts (share of countries across time),

* heatmaps (year vs country intensity matrix).

They support detection of emerging countries or institutions over the years and help identify shifting collaboration patterns.

### (d) Construction logic: The dataset is grouped by year + dimension, and then aggregated by publication counts or distinct partners. Depending on the visual type, the grouped output is reshaped into:

* time series table (for line charts),

* pivot tables (for stacked bars / heatmaps).

## 3.7 Interpretation and user value

The evolution tab provides a higher-level structural reading of collaboration dynamics and allows stakeholders to answer complex questions such as:

* Which centres collaborate with which countries most strongly?

* Are collaborations becoming more concentrated or diversified over time?

* Which centres show the strongest international diversification profiles?

* Which partner institutions are structurally linked to specific teams?

The combination of Sankey (flow structure) and radar (multi-metric profile) yields a complementary analytic approach: the Sankey highlights structural pipelines, while the radar supports comparative performance and diversity analysis.

## 3.8 Technical components

All charts are implemented using interactive plotting libraries that support dynamic filtering and exploration through hover, zoom, and selection tools. The dashboard is conceived as a lightweight front-end layer on top of the enriched Spreadsheet dataset, making it suitable both for on-demand execution and for automated updates (nightly builds).

### Data Profiling Notebook (quality control and performance optimisation)

To ensure the reliability and performance of the dashboard component, an additional data profiling notebook/script was implemented. Its goal is to automatically analyse the final CSV dataset produced by the pipeline and to provide a lightweight diagnostic report on the structure, quality, and consistency of the data. This profiling step is particularly useful before deploying or updating the dashboard, as it helps detect issues such as encoding errors, missing values, duplicated publications, invalid geographic coordinates, and memory-heavy columns that may degrade interactive performance.
The profiling notebook operates as a standalone tool and produces a textual report directly in the console. It was intentionally designed without heavy dependencies (e.g., automated HTML profilers) to remain fast, portable, and easy to run on large datasets.

## Robust loading of the CSV dataset

The first part of the notebook focuses on ensuring that the CSV can be loaded safely, even when it contains inconsistencies. The script defines a CSV input path and uses a custom helper function (load_csv_safely) that implements a two-step reading strategy:

1. Attempt to load the file using UTF-8 encoding,

2. If decoding fails, retry using Latin-1 encoding,

3. In both cases, activate on_bad_lines="skip" to ignore malformed rows.

This approach makes the profiling tool resilient to frequent issues encountered in publication datasets, such as:

* inconsistent character encoding (accented characters),

* corrupted lines due to malformed separators,

* incomplete export rows.

As a result, the profiling tool can still generate a report even when the dataset cannot be perfectly parsed.

## Data type harmonisation (pre-processing step)

Once loaded, the script performs a minimal but important type conversion step to ensure that key variables are consistent with expected dashboard requirements. In particular, it converts:

* the publication year column (Année) into numeric format,

* the geographic coordinates (Latitude, Longitude) into numeric format, using pd.to_numeric/errors="coerce").

Invalid values are automatically coerced into NaN, preventing downstream crashes in interactive charts (time series and map components) and ensuring the dataset behaves correctly during aggregation.

## Global dataset summary (shape and memory footprint)

The profiling report begins with high-level statistics that provide a quick overview of the dataset:

* number of rows and columns,

* list of column names,

* data types (dtype) for each column,

* approximate memory usage in MB.

This step is essential to evaluate whether the dataset is suitable for interactive dashboard usage. It helps identify oversized tables, unexpected column types (e.g., numeric fields stored as strings), and potential optimisation needs for rendering speed profiling_report.

## Missing value analysis

The notebook then performs a missing value analysis by computing the percentage of null values per column:

$$ \text{Missing\%} = \frac{\text{Number of NaN values}}{\text{Total values}} \times 100 $$

The resulting statistics are sorted to highlight the most incomplete columns. This allows the analyst to detect which variables may limit certain visualisations (e.g., missing coordinates affecting the map or missing institutions affecting Sankey flows).

## Column-level exploration (uniqueness and sample values)

For each column, the profiling tool prints:

* the number of unique values,

* the column type,

* up to five sample values.

This step supports semantic validation of categorical fields, such as countries, institutions, city names, centres, and teams. It helps identify inconsistencies in labels (e.g., multiple spellings or mixed formats) and confirms whether variables should be treated as categorical or textual fields in the dashboard.

## Duplicate detection (global and key-based)

The profiling notebook provides duplicate diagnostics by computing:

* the number of fully duplicated rows across all columns (df.duplicated()),

* and, if available, the number of duplicates based on the HAL publication identifier (HalID).

This ensures that publications are not repeated across the dataset, which would artificially inflate publication counts and distort all aggregations in the dashboard (e.g., KPI indicators, top country ranking, time trends)
## Numeric sanity checks (min/max ranges)

For all numerical columns in the dataset, the notebook reports:

* minimum values,

* maximum values.

This allows rapid detection of abnormal values (e.g., invalid years, unrealistic coordinates), and provides a direct validation that numeric fields fall into expected ranges.

## Dashboard-oriented optimisation recommendations

A key feature of this profiling notebook is that it does not merely describe the dataset; it also provides practical recommendations to improve dashboard performance.

### (a) Category conversion for repeated textual fields

The script suggests converting frequently repeated string columns into categorical types (category) such as:

* Centre, Equipe, Ville, Pays,

* Organisme_copubliant,

* UE/Non_UE.

This optimisation reduces memory usage and significantly improves groupby operations and filtering speed, which are critical in interactive dashboards.

### (b) Identification of heavy textual columns

The profiling tool detects potentially large text fields such as:

* addresses (Adresse),

* abstracts (Resume),

and flags them as optional or removable from the dashboard dataset. Keeping such columns can unnecessarily increase dataset size and reduce responsiveness, especially when the dashboard loads the full dataset into memory

### (c) Redundancy detection

The script checks whether two columns contain equivalent information (e.g., Centre and structure) and warns if one is redundant. Removing duplicate columns helps simplify the dataset and avoid storing repeated information.

### (d) Coordinate completeness validation

Finally, it identifies how many records have a city name but missing coordinates (Latitude/Longitude). These cases are crucial because they reduce the completeness of the map visualisation. The notebook quantifies this issue and suggests improving the geocoding step when necessary.

## Final recommendation: optimised storage for interactive use

The script concludes by recommending a standard dashboard optimisation strategy:

* cleaning the dataset,

* saving it in a compressed and efficient format such as Apache Parquet,

* converting key columns to categorical type,

* precomputing frequent aggregations for faster callbacks.

These recommendations ensure that the dashboard remains responsive even on large datasets and that computation is reduced during interactive filtering.

# 4. Performance evaluation

The three scripts are developed in Python (%?), JSON (%?) and TOML (%?). We run the pipeline on a laptop equipped with AMD Ryzen 7 7730U, AMD Radeon™ Onboard Graphics, RAM 16 GB and under WIN11.

The query: *For each Inria Centre: compile a list of foreign co-publishing researchers along with their research institutions, city, and country* was executed in approximately 40 minutes in the case of Centre Inria of Université Côte d'Azur or for Centre Inria Université de Bordeaux. Running the query for all the 9 Inria Centre takes approximately 4 hours. This time can be drastically reduced by retrieving all the data from the API and treating the resulting files off-line, which is part of a future improvement already underway.

90% of the affiliations 'cities, latitudes and longitudes were found. This was verified manually through the Spreadsheet file and through visualisation. We created a dictionary with the validated results that will be used as a means of retrieving cities for successfully treated PIDs in each update.

The results were compared to results obtained earlier by Inria for the same type of queries and proved to me more accurate. One of the reasons was that a copublications included Inria authors from two different Inria Centres, they were counted as copublications for only one of the centres.

## Bibliography

Guha, Kumar, Kazolea, Maria, Liquori, Luigi, Nebot, Andrea and da Silva, Daniel. CoPubli, Software. https://inria.hal.science/hal-05432240, 2025.