National Center for Science and Engineering Statistics

Social, Behavioral and Economic Sciences

National Science Foundation

Building Scientific Publication Profiles 

for U.S.-trained Doctorate Recipients

Haoyi Wei1, Eric Livingston2, Christina Freyman3, and Wan-Ying Chang1

1National Center for Science and Engineering Statistics, National Science Foundation

2Elsevier

3Evaluation and Assessment Capability Section, National Science Foundation
National Center for Science and Engineering Statistics https://ncses.nsf.gov 2

Working papers are intended to report exploratory results of research and 

analysis undertaken by the National Center for Science and Engineering Statistics 

at the National Science Foundation (NSF). Any opinions, findings, conclusions, or 

recommendations expressed in this work do not necessarily reflect the views of 

NSF. This work is being presented to inform interested parties of ongoing 

research or activities and to encourage further discussion of the topic.

Disclaimer
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Brief Summary

Research Goal:

Identify the Scopus publication records for U.S.-trained doctorate recipients in science, 

engineering, and health fields.

Data:

Clarivate’s Survey of Doctorate Recipients (SDR) - Web of Science (WoS) linkage

Elsevier’s SDR – Scopus linkage.

Method:

Machine Learning

Results:

Finalized SDR-Scopus linkage.

Contribution:

This work is an important contribution to the development of a U.S. Science & Engineering 

Enterprise Data Network.

3
National Center for Science and Engineering Statistics https://ncses.nsf.gov 4

- Why do we build publication 

profiles?

- To advance understanding of 

scientific research and the 

impact made by individual 

researchers.

- To investigate the relationships 

between scientific productivity 

and various author attributes.

- To access the nation’s 

strategic investment in doctoral 

training to inform science 

policies. 

Motivation

Citation Bias? Credit: Biomedical Odyssey
Data

Surveys and bibliometric data

5
National Center for Science and Engineering Statistics https://ncses.nsf.gov 6

Survey of Earned Doctorates (SED): annual census of research PhDs

Surveys on U.S.-Trained Doctoral Researchers 

SOURCE: National Center for Science and Engineering Statistics, Survey of Earned Doctorates and Survey of Doctorate Recipients, 2017.
National Center for Science and Engineering Statistics https://ncses.nsf.gov 7

Survey of Doctorate Recipients (SDR): biennial sample survey on science, 

engineering, and health doctorate degree holders

Surveys on U.S.-Trained Doctoral Researchers 

SOURCE: National Center for Science and Engineering Statistics, Survey of Earned Doctorates and Survey of Doctorate Recipients, 2017.
National Center for Science and Engineering Statistics https://ncses.nsf.gov

1st bachelor, master, research PhD, up to 5 

degrees (year, institution, place, field of study); 

dissertation field, financial support, debt

Country/State intend to live, taking a postdoc, 

employment status/commitment, employer 

type, salary, work activities

Sex, marital status, dependents, parent’s 

educational attainment, birthplace, citizenship, 

race/ethnicity, disability

Labor force status, reasons for not working, year 

retired, principal employer, faculty rank, tenure 

status, principal job, work activity, salary, 

benefits, job satisfaction, federal support

Spouse working status, living with children, 

residing location, citizenship/visa type

8

Longitudinal data from SED to SDR

Education

Postgraduation Plans

Background

Employment Situation

Past Employment

Other Work-Related Experiences

Recent Educational Experiences

Demographic Information
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Publication Databases

9

The Web of Science (WoS) and Scopus are 

two leading databases providing reference 

and citation data from academic journals, 

conference proceedings, and other 

documents in various academic disciplines.
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Linkage 1: SDR-WoS

10

Publications

SDR Respondent

Two stages of machine learning 

approaches implemented 

sequentially to address

1. survey-to-publication linking

2. Publication-to-publication 

linking

Publications
National Center for Science and Engineering Statistics https://ncses.nsf.gov 11

Linkage 2: SDR-Scopus (Preliminary)

Author Profiles

SDR Respondent

Publications

All variations of attributes 

from publications within a 

Scopus author profile are 

matched to comparable SDR 

respondent attributes, 

including names, emails, 

affiliation addresses, doctorate 

field, and doctorate year.
Methods

Connecting two independent data linkages

12
National Center for Science and Engineering Statistics https://ncses.nsf.gov

- Connect two independent large scale data linkage to gain 

coverage, data quality, and cost-effectiveness for future 

expansion.

13

Approach

1 + 1 > 2
National Center for Science and Engineering Statistics https://ncses.nsf.gov 14

Digital Object 

Identifier (DOI) is the 

bridge to connect 

matched publications

Join SDR-WoS and SDR-Scopus 
National Center for Science and Engineering Statistics https://ncses.nsf.gov 15

DOI data can effectively confirm Scopus authors

Although DOI is 

known for only half of 

the SDR-WoS 

publications, 83% of 

SDR-WoS authors 

have publication DOI 

data and 78% of SDRWoS authors find 

DOI-confirmed 

Scopus authors in the 

candidate pool.

23%

35%

20%

5%

17%

Confirmed by best match with

DOI

Confirmed only by good match

with DOI

Confirmed only by other match

with DOI

Have DOIs but no confirmed

Scopus author

No DOI

SDR-WoS authors

SOURCE: National Center for Science and Engineering Statistics, Survey of Doctorate Recipients linked to Web of Science Bibliometric Database and Scopus Author Profiles, 2021.
Machine learning

Constructing training and evaluation data

16
National Center for Science and Engineering Statistics https://ncses.nsf.gov 17

- Keep REFID-author ID pairs with confirmed DOIs appeared in high-quality 

SDR-WoS matches.

Construct training data – positive sample
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Construct training data – negative sample

18

- Wrong matches within same refid: remaining (SDR, Scopus author ID) pairs 

under a respondent having a confirm pair with comparable total matched 

publications.

- Non-authors: Candidate pairs under a respondent with no matched publications 

in WoS.
National Center for Science and Engineering Statistics https://ncses.nsf.gov 19

Constructing the training data
National Center for Science and Engineering Statistics https://ncses.nsf.gov 20

Training and evaluation

- Split the training data into training and test sets.

Predictors

- Background and employment outcome: SED and SDR survey data

- Richness of source data for matching: quantity and quality of matching keys

- Scores of similarity: component scores of SOLR query

ML methods tried

- Logistic regression

- Regression tree

- Random Forest

Machine Learning
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Precision and Recall

Precision is the ratio between the correct predictions and the 

total predictions.

Recall is the ratio between the correct predictions and the 

total number of correct items in the set.

21
National Center for Science and Engineering Statistics https://ncses.nsf.gov 22

Evaluate ML predictions

Preliminary results showing 

comparable performance 

among the three ML methods. 

Random Forest model performs 

slightly better in the prediction 

threshold of [0.5, 0.8]

SOURCE: National Center for Science and Engineering Statistics, Survey of Doctorate Recipients linked to Web of Science Bibliometric Database and Scopus Author Profiles, 2021.
Preliminary findings

ML predictions 

23
National Center for Science and Engineering Statistics https://ncses.nsf.gov 24

Match rates: WoS vs. Scopus

Overall match rate: WoS (72%), Scopus (65%)

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Women

Men

NH Black

NH Asian

Hispanic

Other

NH White

Other

Management

Computer application

Teaching

Research &development

psychology

Social sci

Math/Stat

Engineering

Health

Bio/Life

Physical sci

Computer sci

1965 - 19771978 - 19871988 - 19921993 - 19971998 - 20022003 - 2007

2008 +

Sex Race and ethnicity Primary work activity Doctorate field Graduates

WoS Scopus

SOURCE: National Center for Science and Engineering Statistics, Survey of Doctorate Recipients linked to Web of Science Bibliometric Database and Scopus Author Profiles, 2021.
National Center for Science and Engineering Statistics https://ncses.nsf.gov 25

Post-graduation publications: 2008-2012 graduates

Preliminary Scopus matches show similar rate of 

post-PhD publications than SDR-WoS matches. 

Scopus author profiles contain more post-PhD 

publications

Scopus

0%

20%

40%

60%

80%

100%

4-year

universit

y

2-

year/oth

er...

Private,

for profit

Selfemploye

d

Nonprofi

t

Federal

gov

State/Lo

cal gov

Foreign

gov

Not

working

WoS Scopus

SOURCE: National Center for Science and Engineering Statistics, Survey of Doctorate Recipients linked to Web of Science Bibliometric Database and Scopus Author Profiles, 2021.

Post PhD Publications Publication per year

WoS

Scopus
National Center for Science and Engineering Statistics https://ncses.nsf.gov 26

Productivity, collaboration, and impact

At individual author level, more matched 

publication per year, more coauthors, 

more countries of affiliation on average 

are observed from Scopus matches. 

Average 2-year citations are higher from 

WoS matches.

SOURCE: National Center for Science and Engineering Statistics, Survey of Doctorate Recipients linked to Web of Science Bibliometric Database and Scopus Author Profiles, 2021.

Scopus pub per year

WoS pub per year

Scopus avg # of coauthor

WoS avg # of coauthor

Scopus avg 2-year citation

WoS avg 2-year citation

Scopus avg # of county

WoS avg # of county # of pub in Scopus/# of pub in WoS
National Center for Science and Engineering Statistics https://ncses.nsf.gov 27

• ML modeling refinement

• Coverage bias analysis

• Matching quality evaluation

• Linkage updates

• Build use cases

Next steps
28

Contact: Haoyi Wei howei@nsf.gov

Wan-Ying Chang wchang@nsf.gov