Nonstandard Errors
Authors: MENKVELD ALBERT J. , DREBER ANNA , HOLZMEISTER FELIX , HUBER JUERGEN , JOHANNESSON MAGNUS , KIRCHLER MICHAEL , NEUSÜß SEBASTIAN , RAZEN MICHAEL , WEITZEL UTZ , ABAD‐DÍAZ DAVID , ABUDY MENACHEM (MENI) , ADRIAN TOBIAS , AIT‐SAHALIA YACINE , AKMANSOY OLIVIER , ALCOCK JAMIE T. , ALEXEEV VITALI , ALOOSH ARASH , AMATO LIVIA , AMAYA DIEGO , ANGEL JAMES J. , AVETIKIAN ALEJANDRO T. , BACH AMADEUS , BAIDOO EDWIN , BAKALLI GAETAN , BAO LI , BARBON ANDREA , BASHCHENKO OKSANA , BINDRA PARAMPREET C. , BJØNNES GEIR H. , BLACK JEFFREY R. , BLACK BERNARD S. , BOGOEV DIMITAR , CORREA SANTIAGO BOHORQUEZ , BONDARENKO OLEG , BOS CHARLES S. , BOSCH‐ROSA CIRIL , BOURI ELIE , BROWNLEES CHRISTIAN , CALAMIA ANNA , CAO VIET NGA , CAPELLE‐BLANCARD GUNTHER , ROMERO LAURA M. CAPERA , CAPORIN MASSIMILIANO , CARRION ALLEN , CASKURLU TOLGA , CHAKRABARTY BIDISHA , CHEN JIAN , CHERNOV MIKHAIL , CHEUNG WILLIAM , CHINCARINI LUDWIG B. , CHORDIA TARUN , CHOW SHEUNG‐CHI , CLAPHAM BENJAMIN , COLLIARD JEAN‐EDOUARD , COMERTON‐FORDE CAROLE , CURRAN EDWARD , DAO THONG , DARE WALE , DAVIES RYAN J. , BLASIS RICCARDO DE , NARD GIANLUCA F. DE , DECLERCK FANY , DEEV OLEG , DEGRYSE HANS , DEKU SOLOMON Y. , DESAGRE CHRISTOPHE , DIJK MATHIJS A. VAN , DIM CHUKWUMA , DIMPFL THOMAS , DONG YUN JIANG , DRUMMOND PHILIP A. , DUDDA TOM , DUEVSKI TEODOR , DUMITRESCU ARIADNA , DYAKOV TEODOR , DYHRBERG ANNE HAUBO , DZIELIŃSKI MICHAŁ , EKSI ASLI , KALAK IZIDIN EL , ELLEN SASKIA TER , EUGSTER NICOLAS , EVANS MARTIN D. D. , FARRELL MICHAEL , FELEZ‐VINAS ESTER , FERRARA GERARDO , FERROUHI EL MEHDI , FLORI ANDREA , FLUHARTY‐JAIDEE JONATHAN T. , FOLEY SEAN D. V. , FONG KINGSLEY Y. L. , FOUCAULT THIERRY , FRANUS TATIANA , FRANZONI FRANCESCO , FRIJNS BART , FRÖMMEL MICHAEL , FU SERVANNA M. , FÜLLBRUNN SASCHA C. , GAN BAOQING , GAO GE , GEHRIG THOMAS P. , GEMAYEL ROLAND , GERRITSEN DIRK , GIL‐BAZO JAVIER , GILDER DUDLEY , GLOSTEN LAWRENCE R. , GOMEZ THOMAS , GORBENKO ARSENY , GRAMMIG JOACHIM , GRÉGOIRE VINCENT , GÜÇBILMEZ UFUK , HAGSTRÖMER BJÖRN , HAMBUCKERS JULIEN , HAPNES ERIK , HARRIS JEFFREY H. , HARRIS LAWRENCE , HARTMANN SIMON , HASSE JEAN‐BAPTISTE , HAUTSCH NIKOLAUS , HE XUE‐ZHONG (TONY) , HEATH DAVIDSON , HEDIGER SIMON , HENDERSHOTT TERRENCE , HIBBERT ANN MARIE , HJALMARSSON ERIK , HOELSCHER SETH A. , HOFFMANN PETER , HOLDEN CRAIG W. , HORENSTEIN ALEX R. , HUANG WENQIAN , HUANG DA , HURLIN CHRISTOPHE , ILCZUK KONRAD , IVASHCHENKO ALEXEY , IYER SUBRAMANIAN R. , JAHANSHAHLOO HOSSEIN , JALKH NAJI , JONES CHARLES M. , JURKATIS SIMON , JYLHÄ PETRI , KAECK ANDREAS T. , KAISER GABRIEL , KARAM ARZÉ , KARMAZIENE EGLE , KASSNER BERNHARD , KAUSTIA MARKKU , KAZAK EKATERINA , KEARNEY FEARGHAL , KERVEL VINCENT VAN , KHAN SAAD A. , KHOMYN MARTA K. , KLEIN TONY , KLEIN OLGA , KLOS ALEXANDER , KOETTER MICHAEL , KOLOKOLOV ALEKSEY , KORAJCZYK ROBERT A. , KOZHAN ROMAN , KRAHNEN JAN P. , KUHLE PAUL , KWAN AMY , LAJAUNIE QUENTIN , LAM F. Y. ERIC C. , LAMBERT MARIE , LANGLOIS HUGUES , LAUSEN JENS , LAUTER TOBIAS , LEIPPOLD MARKUS , LEVIN VLADIMIR , LI YIJIE , LI HUI , LIEW CHEE YOONG , LINDNER THOMAS , LINTON OLIVER , LIU JIACHENG , LIU ANQI , LLORENTE GUILLERMO , LOF MATTHIJS , LOHR ARIEL , LONGSTAFF FRANCIS , LOPEZ‐LIRA ALEJANDRO , MANKAD SHAWN , MANO NICOLA , MARCHAL ALEXIS , MARTINEAU CHARLES , MAZZOLA FRANCESCO , MELOSO DEBRAH , MI MICHAEL G. , MIHET ROXANA , MOHAN VIJAY , MOINAS SOPHIE , MOORE DAVID , MU LIANGYI , MURAVYEV DMITRIY , MURPHY DERMOT , NESZVEDA GABOR , NEUMEIER CHRISTIAN , NIELSSON ULF , NIMALENDRAN MAHENDRARAJAH , NOLTE SVEN , NORDEN LARS L. , O'NEILL PETER , OBAID KHALED , ØDEGAARD BERNT A. , ÖSTBERG PER , PAGNOTTA EMILIANO , PAINTER MARCUS , PALAN STEFAN , PALIT IMON J. , PARK ANDREAS , PASCUAL ROBERTO , PASQUARIELLO PAOLO , PASTOR LUBOS , PATEL VINAY , PATTON ANDREW J. , PEARSON NEIL D. , PELIZZON LORIANA , PELLI MICHELE , PELSTER MATTHIAS , PÉRIGNON CHRISTOPHE , PFIFFER CAMERON , PHILIP RICHARD , PLÍHAL TOMÁŠ , PRAKASH PUNEET , PRESS OLIVER‐ALEXANDER , PRODROMOU TINA , PROKOPCZUK MARCEL , PUTNINS TALIS , QIAN YA , RAIZADA GAURAV , RAKOWSKI DAVID , RANALDO ANGELO , REGIS LUCA , REITZ STEFAN , RENAULT THOMAS , RENJIE REX W. , RENO ROBERTO , RIDDIOUGH STEVEN J. , RINNE KALLE , RINTAMÄKI PAUL , RIORDAN RYAN , RITTMANNSBERGER THOMAS , LONGARELA IÑAKI RODRÍGUEZ , ROESCH DOMINIK , ROGNONE LAVINIA , ROSEMAN BRIAN , ROŞU IOANID , ROY SAURABH , RUDOLF NICOLAS , RUSH STEPHEN R. , RZAYEV KHALADDIN , RZEŹNIK ALEKSANDRA A. , SANFORD ANTHONY , SANKARAN HARIKUMAR , SARKAR ASANI , SARNO LUCIO , SCAILLET OLIVIER , SCHARNOWSKI STEFAN , SCHENK‐HOPPÉ KLAUS R. , SCHERTLER ANDREA , SCHNEIDER MICHAEL , SCHROEDER FLORIAN , SCHÜRHOFF NORMAN , SCHUSTER PHILIPP , SCHWARZ MARCO A. , SEASHOLES MARK S. , SEEGER NORMAN J. , SHACHAR OR , SHKILKO ANDRIY , SHUI JESSICA , SIKIC MARIO , SIMION GIORGIA , SMALES LEE A. , SÖDERLIND PAUL , SOJLI ELVIRA , SOKOLOV KONSTANTIN , SÖNKSEN JANTJE , SPOKEVICIUTE LAIMA , STEFANOVA DENITSA , SUBRAHMANYAM MARTI G. , SZASZI BARNABAS , TALAVERA OLEKSANDR , TANG YUEHUA , TAYLOR NICK , THAM WING WAH , THEISSEN ERIK , THIMME JULIAN , TONKS IAN , TRAN HAI , TRAPIN LUCA , TROLLE ANDERS B. , VADUVA M. ANDREEA , VALENTE GIORGIO , NESS ROBERT A. VAN , VASQUEZ AURELIO , VEROUSIS THANOS , VERWIJMEREN PATRICK , VILHELMSSON ANDERS , VILKOV GRIGORY , VLADIMIROV VLADIMIR , VOGEL SEBASTIAN , VOIGT STEFAN , WAGNER WOLF , WALTHER THOMAS , WEISS PATRICK , WEL MICHEL VAN DER , WERNER INGRID M. , WESTERHOLM P. JOAKIM , WESTHEIDE CHRISTIAN , WIKA HANS C. , WIPPLINGER EVERT , WOLF MICHAEL , WOLFF CHRISTIAN C. P. , WOLK LEONARD , WONG WING‐KEUNG , WRAMPELMEYER JAN , WU ZHEN‐XING , XIA SHUO , XIU DACHENG , XU KE , XU CAIHONG , YADAV PRADEEP K. , YAGÜE JOSÉ , YAN CHENG , YANG ANTTI , YOO WOONGSUN , YU WENJIA , YU YIHE , YU SHIHAO , YUESHEN BART Z. , YUFEROVA DARYA , ZAMOJSKI MARCIN , ZAREEI ABALFAZL , ZEISBERGER STEFAN M. , ZHANG LU , ZHANG S. SARAH , ZHANG XIAOYU , ZHAO LU , ZHONG ZHUO , ZHOU Z. IVY , ZHOU CHEN , ZHU XINGYU S. , ZOICAN MARIUS , ZWINKELS REMCO 
Published: April 2024

ABSTRACTIn statistics, samples are drawn from a population in a data‐generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence‐generating process (EGP). We claim that EGP variation across researchers adds uncertainty—nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer‐review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.