<figure><img src="image_1.png" /></figure>

HAL
open science

# Designing a sample and survey management service to support quality methods for the ESS CRONOS 2 online panel

Malaury Lemaître-Salmon

► To cite this version:

Malaury Lemaître-Salmon. Designing a sample and survey management service to support quality methods for the ESS CRONOS 2 online panel. ESRA 2023 (European Survey Research Association), European Survey Research Association, Jul 2023, Milan, Italy. **hal-04169478**

HAL Id: hal-04169478

https://sciencespo.hal.science/hal-04169478

Submitted on 24 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Designing a sample and survey management service to support quality methods for the ESS CRONOS 2 online panel

Malaury Lemaître-Salmon, CDSP (Sciences Po, CNRS)
**European Survey Research Association Conference**
21-07-2023

funded by the EC under SSHOC & ESS-SUSTAIN-2 (grants #823782 and #871063)

<figure><img src="image_3.png" /></figure>
The project

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

**SSHOC**
social sciences & humanities open cloud
Technical Stack

<figure><img src="image_6.png" /></figure>

python™

django

<figure><img src="image_7.png" /></figure>

CELERY

<figure><img src="image_8.png" /></figure>

PostgreSQL

<figure><img src="image_9.png" /></figure>

Bootstrap

<figure><img src="image_10.png" /></figure>

kubernetes

<figure><img src="image_11.png" /></figure>

docker
# What is this service for?
Securing personal data

Minimized data flows.

Decentralised and privacy-compliant
handling of panelist data.

User roles defined on a need-to-know
basis.

<figure><img src="image_12.png" /></figure>

icon credit: Hat-Tech, Noun project
# Collaboration & orchestration

Centralised management of survey fielding orchestration:

- content edition monitoring

- questionnaire and message edition

- questionnaire and messages translation

- fieldwork coordination

- questionnaire publication

- sending invites & reminders

<figure><img src="image_13.png" /></figure>
Flexible contact modes

Messages can be send to panelists by email
or short text messages.

The two modes of contact can be
intertwined.

<figure><img src="image_14.png" /></figure>

icon credit: K. Ginkaew, Noun project
<action triggers>

WPSS

headquarters / survey coordinator

national coordinators / sample managers

(respondents)

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

API

<content edition>

Qualtrics®

survey developers

messages editors & translators

data archive staff

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>
# WPSS Features

<figure><img src="image_22.png" /></figure>
<figure><img src="image_23.png" /></figure>

# Headquarters

Entry point for headquarters managers

## Panelists

<figure><img src="image_24.png" /></figure>

View

## Panels

<figure><img src="image_25.png" /></figure>

View

## National Coordinators

<figure><img src="image_26.png" /></figure>

View

## Surveys

<figure><img src="image_27.png" /></figure>

View

## Individual links

<figure><img src="image_28.png" /></figure>

View

## Message deliveries

<figure><img src="image_29.png" /></figure>

View
<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

Manage additional variables

Summary of language codes: 'FR' 'IT'

Filters

Reset

29/29 panelists

<table><thead><tr><th>ID ▲▼</th><th>Age ▲▼</th><th>Sex ▲▼</th><th>Language ▲▼</th><th>Email ▲▼</th><th>Phone ▲▼</th><th>Address ▲▼</th><th>Internet use ▲▼</th><th>Opt out ▲▼</th><th>Opt out date ▲▼</th><th>Opt out reason ▲▼</th><th>Delete contact data ▲▼</th><th>Delete survey data ▲▼</th><th>Out of country ▲▼</th><th>No incentive ▲▼</th><th>No letter ▲▼</th><th>No text ▲▼</th></tr></thead><tbody><tr><td>FR0</td><td>50</td><td>Male</td><td>FR</td><td>✓</td><td>✗</td><td>✗</td><td>Never</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR1</td><td>30</td><td>Female</td><td>FR</td><td>✓</td><td>✗</td><td>✗</td><td>Only occasionally</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR2</td><td>47</td><td>Other</td><td>FR</td><td>✓</td><td>✗</td><td>✗</td><td>A few times a week</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR3</td><td>45</td><td>Refusal</td><td>FR</td><td>✓</td><td>✗</td><td>✗</td><td>Most days</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR4</td><td>45</td><td>Don't know</td><td>FR</td><td>✓</td><td>✓</td><td>✗</td><td>Every day</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR5</td><td>44</td><td>No answer</td><td>FR</td><td>✓</td><td>✓</td><td>✗</td><td>Refusal</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR6</td><td>46</td><td>Male</td><td>FR</td><td>✓</td><td>✓</td><td>✗</td><td>Don't know</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR7</td><td>29</td><td>Female</td><td>FR</td><td>✓</td><td>✓</td><td>✗</td><td>No answer</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR8</td><td>46</td><td>Other</td><td>FR</td><td>✗</td><td>✓</td><td>✗</td><td>Never</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR9</td><td>29</td><td>Refusal</td><td>FR</td><td>✗</td><td>✓</td><td>✗</td><td>Only occasionally</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR10</td><td>36</td><td>Don't know</td><td>FR</td><td>✗</td><td>✓</td><td>✗</td><td>A few times a week</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR11</td><td>44</td><td>No answer</td><td>FR</td><td>✗</td><td>✓</td><td>✗</td><td>Most days</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR12</td><td>50</td><td>Male</td><td>FR</td><td>✗</td><td>✓</td><td>✗</td><td>Every day</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR13</td><td>43</td><td>Female</td><td>FR</td><td>✓</td><td>✓</td><td>✗</td><td>Refusal</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>FR14</td><td>47</td><td>Other</td><td>FR</td><td>✓</td><td>✓</td><td>✗</td><td>Don't know</td><td>✗</td><td>None</td><td></td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr></tbody></table>

<figure><img src="image_32.png" /></figure>
Headquarters →

# New set of individual links

Select a survey and targeted panels, for which individual links will be generated when appropriate

As a first step, the form below will allow you to select a survey and target panels, a give a short and meaningful description of the intended use of this link set.
No links will be generated yet.

Generating the links will come as a separate step, at which point you will be asked to specify the link's expiry date. This can be done from the links set's detail page.

Panels *

Panel_1

Panel_2

Hold down the SHIFT key while clicking to select several panels

Description *

First link

A concise way to unambiguously refer to this link set

Survey *

Samplesurvey_2023

Eligible surveys are those that are active and shared with the WPSS Qualtrics user.

Create

Cancel
<figure><img src="image_33.png" /></figure>

Headquarters → Link sets → Detail →

Generate links

<table><tr><td>UID</td><td>bf8e7c54</td></tr><tr><td>Description</td><td>First link</td></tr><tr><td>Survey</td><td>Samplesurvey_2023</td></tr><tr><td>Panels</td><td><ul><li>Panel_1</li><li>Panel_2</li></ul></td></tr></table>

Expiration date *

<figure><img src="image_34.png" /></figure>

YYYY-MM-DD

Generate

Cancel
<figure><img src="image_35.png" /></figure>

Headquarters → Message deliveries →

## Create a message delivery

Deliver a set of individual links to their recipients using an email or SMS drafted and translated on Qualtrics. A fallback message can optionally be added.

Link set *

(b593d7dd) second_test

<figure><img src="image_36.png" /></figure>

Target recipients *

All

<figure><img src="image_37.png" /></figure>

Message *

Super invite email

<figure><img src="image_38.png" /></figure>

From Qualtrics library

Subject *

Super subject message

<figure><img src="image_39.png" /></figure>

From Qualtrics library

Create

Headquarters → Message deliveries →

## Message delivery details

Detailed summary of this message delivery's settings

UID f1555fae

Type Email

Target All

Link set second_test

Subject Super subject message

Message Super invite email

Add fallback message

Send

Delete
<figure><img src="image_40.png" /></figure>

Headquarters → Link sets →

## Details

Detailed summary of this link set's characteristics

<table><tr><td>UID</td><td>b593d7dd</td></tr><tr><td>Description</td><td>second_test</td></tr><tr><td>Survey</td><td>TestSurvey</td></tr><tr><td>Number of links</td><td>29</td></tr><tr><td>Generated at</td><td>2023-06-28 14h05</td></tr><tr><td>Expire in</td><td>2 months, 3 weeks</td></tr><tr><td>In progress</td><td>10</td></tr><tr><td>Finished</td><td>5</td></tr></table>

## Detailed stats

<table><thead><tr><td>Panel</td><td>Not Started</td><td>Started</td><td>Partially Finished</td><td>Finished</td><td>Total</td></tr></thead><tbody><tr><td>Panel_2</td><td>10</td><td>3</td><td>0</td><td>2</td><td>15</td></tr><tr><td>Panel_1</td><td>4</td><td>7</td><td>0</td><td>3</td><td>14</td></tr><tr><td>Total</td><td>14</td><td>10</td><td>0</td><td>5</td><td>29</td></tr></tbody></table>
<figure><img src="image_41.png" /></figure>

Headquarters → Message deliveries →

Message delivery details

Detailed summary of this message delivery's settings

<table><tr><td>UID</td><td>f1bbfef2</td></tr><tr><td>Type</td><td>Email</td></tr><tr><td>Target</td><td>All</td></tr><tr><td>Link set</td><td>second_test</td></tr><tr><td>Subject</td><td>Super subject message</td></tr><tr><td>Message</td><td>Super invite email</td></tr><tr><td>Send date</td><td>2023-06-28 14h07</td></tr><tr><td>Fallback</td><td>(9d9bc39e) nouveau sms</td></tr></table>

Stats

<table><thead><tr><td>Panel</td><td>Current Panelists</td><td>Attempted</td><td>Delivered</td><td>Opened (at least)</td><td>Soft Bounce</td><td>Hard Bounce</td><td>Other Failures</td></tr></thead><tbody><tr><td>Panel_2</td><td>15</td><td>10</td><td>8</td><td>4</td><td>1</td><td>1</td><td>0</td></tr><tr><td>Panel_1</td><td>14</td><td>9</td><td>9</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Total</td><td>29</td><td>19</td><td>17</td><td>4</td><td>1</td><td>1</td><td>0</td></tr></tbody></table>
<figure><img src="image_42.png" /></figure>

National coordinators → Panels →

Panel_2

Panel information

Panelists

<figure><img src="image_43.png" /></figure>

View

Import history

<figure><img src="image_44.png" /></figure>

View

Individual Links

<figure><img src="image_45.png" /></figure>

View

Message deliveries

<figure><img src="image_46.png" /></figure>

View

Settings

<figure><img src="image_47.png" /></figure>

View
<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>

Summary of language codes: 'FR'

Filters Reset

15/15 panelists

<table><thead><tr><th>ID</th><th>▲▼</th><th>First name</th><th>▲▼</th><th>Last name</th><th>▲▼</th><th>Sex</th><th>▲▼</th><th>Email</th><th>▲▼</th><th>Phone</th><th>▲▼</th><th>Language</th><th>▲▼</th><th>State</th><th>Additional variables</th><th>Edit</th></tr></thead><tbody><tr><td>FR0</td><td></td><td>Anthony</td><td></td><td>Phillips</td><td></td><td>Male</td><td></td><td>panelist_20@qxsms.com</td><td></td><td>None</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR1</td><td></td><td>Angela</td><td></td><td>Castillo</td><td></td><td>Female</td><td></td><td>panelist_21@qxsms.com</td><td></td><td>None</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR2</td><td></td><td>Heather</td><td></td><td>Rosales</td><td></td><td>Other</td><td></td><td>panelist_22@qxsms.com</td><td></td><td>None</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR3</td><td></td><td>Matthew</td><td></td><td>Wyatt</td><td></td><td>Refusal</td><td></td><td>panelist_23@qxsms.com</td><td></td><td>None</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR4</td><td></td><td>William</td><td></td><td>Gray</td><td></td><td>Don't know</td><td></td><td>panelist_24@qxsms.com</td><td></td><td>+380977327312</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR5</td><td></td><td>Cory</td><td></td><td>Reyes</td><td></td><td>No answer</td><td></td><td>panelist_25@qxsms.com</td><td></td><td>+36318869465</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR6</td><td></td><td>Dawn</td><td></td><td>Williams</td><td></td><td>Male</td><td></td><td>panelist_26@qxsms.com</td><td></td><td>+38644599298</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR7</td><td></td><td>Robert</td><td></td><td>Brown</td><td></td><td>Female</td><td></td><td>panelist_27@qxsms.com</td><td></td><td>+380621079598</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR8</td><td></td><td>Debbie</td><td></td><td>Sanders</td><td></td><td>Other</td><td></td><td>None</td><td></td><td>+37167799788</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR9</td><td></td><td>Christopher</td><td></td><td>Patterson</td><td></td><td>Refusal</td><td></td><td>None</td><td></td><td>+38533589865</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR10</td><td></td><td>Sarah</td><td></td><td>Ramirez</td><td></td><td>Don't know</td><td></td><td>None</td><td></td><td>+33479671936</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR11</td><td></td><td>Jennifer</td><td></td><td>Lara</td><td></td><td>No answer</td><td></td><td>None</td><td></td><td>+17405923612</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR12</td><td></td><td>Steven</td><td></td><td>Curtis</td><td></td><td>Male</td><td></td><td>None</td><td></td><td>+33355236302</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR13</td><td></td><td>Alison</td><td></td><td>Thompson</td><td></td><td>Female</td><td></td><td>panelist_33@qxsms.com</td><td></td><td>+306934707229</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr><tr><td>FR14</td><td></td><td>Joshua</td><td></td><td>Wiley</td><td></td><td>Other</td><td></td><td>panelist_34@qxsms.com</td><td></td><td>+38532828915</td><td></td><td>FR</td><td></td><td>Never logged in</td><td></td><td>✓</td></tr></tbody></table>

<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>
## WPSS key points

- Transparency of process

- Complete control of the study

- Precise synchronisation of all process

- Effective internationalization

- GDPR principles strictly implemented

- Contact mode flexibility

- Panelist portal availability

- Comprehensive support
## Critical feedback

WPSS would require substantial rewriting to work with a survey platform other than Qualtrics.

The integration of Qualtrics was necessary to meet project deadlines and due to its widespread use in the academic community.

The use of the Qualtrics API presents challenges, including a lack of precise support and unexpected changes.

Errors with the Qualtrics server and support issues were encountered.
Thank you

[itcdsp-scpolst@sciencespo.fr](mailto:itcdsp-scpolst@sciencespo.fr)

[malaury.lemaitresalmon@sciencespo.fr](mailto:malaury.lemaitresalmon@sciencespo.fr)