Panel on Reproducibility and Transparency

Transparency of 

Software and Code

Lars Vilhuber
Benefits of Transparency and 

Reproducibility

• efficiency,

• innovation and 

progress, 

• trust and 

confidence, 

• and the value from 

the use of the data 

products

National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National 

Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National 

Academies Press. https://doi.org/10.17226/26360.
Transparency and: Proprietary Software

Transparency 

requires

“the provision of 

sufficiently 

detailed 

documentation”
Transparency and: Proprietary Software

Transparency 

requires

“the provision of 

sufficiently 

detailed 

documentation”

Blaise LimeSurvey

Qualtrics

SAS

Python

R

Oracle

PostgreSQL
Transparency and: Proprietary Software

Transparency 

requires

“the provision of 

sufficiently 

detailed 

documentation”

Q: What to do when 

software is 

proprietary?
Transparency and: Proprietary Software

Transparency 

requires

“the provision of 

sufficiently 

detailed 

documentation”

Q: What to do when 

software is 

proprietary?

A: Clearly describe 

use of software

(accessibility, price, 

version)
Transparency and: Code

Code is

instructions to 

make software 

function

(could also be source code for software)

- functionality of 

code

- development of 

code (when, who)

- dependencies
Transparency and: Code

Code is

instructions to 

make software 

function

(could also be source code for software)
Transparency and: Code

- functionality of 

code
Transparency and: Code

- functionality of 

code

- development of 

code (when, who)

- Includes release policy
Transparency and: Code

- functionality of 

code

- development of 

code (when, who)

- dependencies
Transparency and Software

Recommendation 4.1

Agencies that produce federal statistics [...] 

- should review and make a priority of

- adopting modern information technology tools that 

assist in 

- collaborative software development and 

documentation of workflow and methodology.
Transparency and Software

Company X

Recommendation 4.1

- Use versioning systems

- Use them broadly (not 

just selectively)

- Create and use 

style guides

- Not just for 

“developers”
Some particular notes
Transparency and: Surveys

Agencies have 

exemplary tradition 

of publishing 

questionnaires

- Most often not in 

re-usable formats 

(such as DDI, Blaise)

- Transparency of 

“code” would allow 

greater 

reuse/consistency
Example: Survey of Earned Doctorates

“Data collection. In 2020, for the first 

time, the SED data collection did not use 

the self-administered paper questionnaire. 

The SED was completed primarily by selfadministered Web survey with a small 

number of nonrespondents contacted to 

complete computer-assisted telephone 

interviewing (CATI).”

https://ncses.nsf.gov/pubs/nsf22300/technical-notes#survey-design
Transparency and: Processing

All data cleaning 

and preparation is 

(or should be)

done by code

- Can such code be 

made available?

- The processed data is 

probably still 

confidential (PII)
Analogy: Code for academic articles

All data cleaning and 

preparation code 

(and instructions) 

MUST be provided.
Do’s and don’t’s
Do’s and don’t’s
Do’s and don’t’s
Do’s and don’t’s
Do’s and don’t’s

• lars.vilhuber.com/p/fsrdc2022/
Transparency and: Processing

Tracing of code 

execution is hard 

(logging)

- Already occurs for 

security purposes

- May not need to be at 

the finest level for 

transparency

- May not need to be 

publicly available (but 

auditable)
Transparency and: Processing

Tracing of code 

execution is hard 

(logging)

- Already occurs for 

security purposes

- May not need to be at 

the finest level for 

transparency

- May not need to be 

publicly available (but 

auditable)
Transparency and: Manual steps

Some manual steps

in processing may 

be unavoidable

Transparency implies

- that it be 

identified

- documented 

(instructions, 

training manuals)

- ideally publicly
Transparency and: Manual steps

Some manual steps

in processing may 

be unavoidable

Can include:

• Rules for manual 

edits

• Human edits
Transparency and: Consequences

Transparency can 

be hard

• 1,000 of people 

looking “over your 

shoulder”

• Errors will be 

found
Transparency and: Consequences

Transparency can 

be hard

... but valuable

• Self-disciplining 

device

• Possible crowdsourcing of 

solutions
Transparency and: Policies

Transparency 

needs frameworks

• Internal policies on 

how to respond to 

(legitimate) 

criticism

• Support for the 

process
Transparency and: Policies

Transparency 

needs frameworks

• Coding guides

• Continuous review 

process

– For quality

– For security

• Training!
doi.org/10.17226/26360