<figure><img src="image_355.png" /></figure>

MrPhi

@MonsieurPhi

...

Petite note: j'ai remarqué que jusqu'à 1800 Elo, sa sensibilité au prompt est un peu bizarre. Par exemple avec "1-0" (Blanc gagne) il joue le coup correct g6. Mais avec "0-1" (Noir gagne) ou "1/2-1/2", il joue Nf6 (ce qui est illogique vu que Nf6 fait justement perdre les Noirs).

Translate post

<figure><img src="image_356.png" /></figure>

11:55 AM · Apr 19, 2024 · 447 Views

<figure><img src="image_357.png" /></figure>

<figure><img src="image_358.png" /></figure>

<figure><img src="image_359.png" /></figure>

<figure><img src="image_360.png" /></figure>

<figure><img src="image_361.png" /></figure>

<figure><img src="image_362.png" /></figure>

Post your reply

Reply

<figure><img src="image_363.png" /></figure>

MrPhi @MonsieurPhi · Apr 19

Mais il n'a plus ce comportement bizarre quand le Elo dépasse 2000. Enfin bref ! Il faudrait faire d'autres tests pour déterminer à quel point cette information sur le niveau Elo influence son niveau de jeu, mais c'est déjà intéressant comme petite expérience.

<figure><img src="image_364.png" /></figure>

<figure><img src="image_365.png" /></figure>

<figure><img src="image_366.png" /></figure>

<figure><img src="image_367.png" /></figure>

<figure><img src="image_368.png" /></figure>

<figure><img src="image_369.png" /></figure>

<figure><img src="image_370.png" /></figure>

<figure><img src="image_371.png" /></figure>
<figure><img src="image_339.png" /></figure>

MrPhi

@MonsieurPhi

...

Petite note: j'ai remarqué que jusqu'à 1800 Elo, sa sensibilité au prompt est un peu bizarre. Par exemple avec "-1-0" (Blanc gagne) il joue le coup correct g6. Mais avec "-0-1" (Noir gagne) ou "-1/2-1/2", il joue Nf6 (ce qui est illogique vu que Nf6 fait justement perdre les Noirs).

Translate post

<figure><img src="image_340.png" /></figure>

11:55 AM - Apr 19, 2024

<figure><img src="image_341.png" /></figure>

447 Views

<figure><img src="image_342.png" /></figure>

<figure><img src="image_343.png" /></figure>

<figure><img src="image_344.png" /></figure>

<figure><img src="image_345.png" /></figure>

<figure><img src="image_346.png" /></figure>

<figure><img src="image_347.png" /></figure>

Post your reply

Reply

<figure><img src="image_348.png" /></figure>

MrPhi @MonsieurPhi - Apr 19

Mais il n'a plus ce comportement bizarre quand le Elo dépassé 2000. Enfin bref ! Il faudrait faire d'autres tests pour déterminer à quel point cette information sur le niveau Elo influence son niveau de jeu, mais c'est déjà intéressant comme petite expérience.

<figure><img src="image_349.png" /></figure>

<figure><img src="image_350.png" /></figure>

<figure><img src="image_351.png" /></figure>

<figure><img src="image_352.png" /></figure>

<figure><img src="image_353.png" /></figure>

# Optional GM titles

black_title = "[BlackTitle \

white_title = "[WhiteTitle \

# Construct the PGN header with the configurable options

pgn_headers = f"""[Event "FIDE World Championship Match 2024"]

[Site "Los Angeles, USA"]

[Date "2024.12.01"]

[Round "5"]

[White "{white_name}"]

[Black "{black_name}"]

[Result "{result}"]

[WhiteElo "{white elo}"]

{white_title}

[BlackElo "{black elo}"]

[black_title]

[TimeControl "40/7200:20/3600:900+30"]

[UTCDate "2024.11.27"]

[UTCTime "09:01:25"]

[Variant "Standard"]

"""

# Define possible values for each parameter

results = ["1-0", "1/2-1/2", "0-1"]

# results = ["1-0", "0-1"]

names = ["Nepomniachtchi, Ian", "Kramnik, Vladimir", "Giraud, Thibaut", "Louapre, David", "XXX"]

elos = [1000, 1400, 1700, 1800, 2000, 2900]

# include_title = [True, False]

include_title = [False]

gpt_config = GPTConfig(
    model_gpt="gpt-3.5-turbo-instruct",
    temperature=0.0,
    max_tokens=5,
    chat_gpt=False,
    system_role_message=None  # Since it wasn't provided in the original call
)

<figure><img src="image_354.png" /></figure>

<figure><img src="image_355.png" /></figure>
<figure><img src="image_372.png" /></figure>

MrPhi

@MonsieurPhi

...

Petite note: j'ai remarqué que jusqu'à 1800 Elo, sa sensibilité au prompt est un peu bizarre. Par exemple avec "-1-0" (Blanc gagne) il joue le coup correct g6. Mais avec "-0-1" (Noir gagne) ou "1/2-1/2", il joue Nf6 (ce qui est illogique vu que Nf6 fait justement perdre les Noirs).

Translate point

<figure><img src="image_373.png" /></figure>

11:55 AM - Apr 19, 2024

<figure><img src="image_374.png" /></figure>

447

Views

<figure><img src="image_375.png" /></figure>

<figure><img src="image_376.png" /></figure>

<figure><img src="image_377.png" /></figure>

<figure><img src="image_378.png" /></figure>

<figure><img src="image_379.png" /></figure>

<figure><img src="image_380.png" /></figure>

Post your reply

Reply

<figure><img src="image_381.png" /></figure>

MrPhi @MonsieurPhi - Apr 19

Mais il n'a plus ce comportement bizarre quand le Elo dépasse 2000. Enfin bref ! Il faudrait faire d'autres tests pour déterminer à quel point cette information sur le niveau Elo influence son niveau de jeu, mais c'est déjà intéressant comme petite expérience.

<figure><img src="image_382.png" /></figure>

<figure><img src="image_383.png" /></figure>

<figure><img src="image_384.png" /></figure>

<figure><img src="image_385.png" /></figure>

<figure><img src="image_386.png" /></figure>

# Optional GM titles

black_title = "[BlackTitle \

white_title = "[WhiteTitle \

# Construct the PGN header with the configurable options
pgn_headers = f"""[Event "FIDE World Championship Match 2024"]

[Site "Los Angeles, USA"]

[Date "2024.12.01"]

[Round "5"]

[White "{white_name}"]

[Black "{black_name}"]

[Result "{result}"]

[WhiteElo "{white elo}"]

{white_title}

[BlackElo "{black elo}"]

[black_title]

[TimeControl "40/7200:20/3600:900+30"]

[UTCDate "2024.11.27"]

[UTCTime "09:01:25"]

[Variant "Standard"]

"""

# Define possible values for each parameter

results = ["1-0", "1/2-1/2", "0-1"]

# results = ["1-0", "0-1"]

names = ["Nepomniachtchi, Ian", "Kramnik, Vladimir", "Giraud, Thibaut", "Louapre, David", "XXX"]

elos = [1000, 1400, 1700, 1800, 2000, 2900]

# include_title = [True, False]

include_title = [False]

gpt_config = GPTConfig(
    model_gpt="gpt-3.5-turbo-instruct",
    temperature=0.0,
    max_tokens=5,
    chat_gpt=False,
    system_role_message=None  # Since it wasn't provided in the original call
)

<figure><img src="image_387.png" /></figure>

<figure><img src="image_388.png" /></figure>
<figure><img src="image_356.png" /></figure>

MrPhi

@MonsieurPhi

...

Petite note : j'ai remarqué que jusqu'à 1800 Elo, sa sensibilité au prompt est un peu bizarre. Par exemple avec "1-0" (Blanc gagne) il joue le coup correct g6. Mais avec "0-1" (Noir gagne) ou "1/2-1/2", il joue Nf6 (ce qui est illogique vu que Nf6 fait justement perdre les Noirs).

Translate post

<figure><img src="image_357.png" /></figure>

11:55 AM · Apr 19, 2024 · 447 Views

<figure><img src="image_358.png" /></figure>

1

<figure><img src="image_359.png" /></figure>

<figure><img src="image_360.png" /></figure>

<figure><img src="image_361.png" /></figure>

<figure><img src="image_362.png" /></figure>

<figure><img src="image_363.png" /></figure>

Post your reply

Reply

<figure><img src="image_364.png" /></figure>

MrPhi @MonsieurPhi · Apr 19

Mais il n'a plus ce comportement bizarre quand le Elo dépasse 2000. Enfin bref ! Il faudrait faire d'autres tests pour déterminer à quel point cette information sur le niveau Elo influence son niveau de jeu, mais c'est déjà intéressant comme petite expérience.

<figure><img src="image_365.png" /></figure>

<figure><img src="image_366.png" /></figure>

<figure><img src="image_367.png" /></figure>

<figure><img src="image_368.png" /></figure>

441

<figure><img src="image_369.png" /></figure>

<figure><img src="image_370.png" /></figure>

<table><thead><tr><th>Result</th><th>WhiteName</th><th>BlackName</th><th>WhiteElo</th><th>BlackElo</th><th>IncludeWhiteTitle</th><th>IncludeBlackTitle</th><th>Move</th></tr></thead><tbody><tr><td>0-1</td><td>XXX</td><td>Louapre, David</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>Nf6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>XXX</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Louapre, David</td><td>XXX</td><td>1400</td><td>1400</td><td>False</td><td>False</td><td>Nf6</td></tr><tr><td>1-0</td><td>XXX</td><td>Nepomniachtchi, Ian</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>Nepomniachtchi, Ian</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Louapre, David</td><td>Nepomniachtchi, Ian</td><td>1000</td><td>1000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Nepomniachtchi, Ian</td><td>XXX</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>XXX</td><td>Nepomniachtchi, Ian</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>XXX</td><td>1700</td><td>1700</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>XXX</td><td>1700</td><td>1700</td><td>False</td><td>False</td><td>Nf6</td></tr></tbody></table>

<figure><img src="image_371.png" /></figure>

<figure><img src="image_372.png" /></figure>

<figure><img src="image_373.png" /></figure>
<figure><img src="image_389.png" /></figure>

MrPhi

@MonsieurPhi

...

Petite note : j'ai remarqué que jusqu'à 1800 Elo, sa sensibilité au prompt est un peu bizarre. Par exemple avec "1-0" (Blanc gagne) il joue le coup correct g6. Mais avec "0-1" (Noir gagne) ou "1/2-1/2", il joue Nf6 (ce qui est illogique vu que Nf6 fait justement perdre les Noirs).

Translate post

<figure><img src="image_390.png" /></figure>

11:55 AM · Apr 19, 2024 · 447 Views

<figure><img src="image_391.png" /></figure>

1

<figure><img src="image_392.png" /></figure>

<figure><img src="image_393.png" /></figure>

<figure><img src="image_394.png" /></figure>

<figure><img src="image_395.png" /></figure>

<figure><img src="image_396.png" /></figure>

Post your reply

Reply

<figure><img src="image_397.png" /></figure>

MrPhi @MonsieurPhi · Apr 19

Mais il n'a plus ce comportement bizarre quand le Elo dépasse 2000. Enfin bref ! Il faudrait faire d'autres tests pour déterminer à quel point cette information sur le niveau Elo influence son niveau de jeu, mais c'est déjà intéressant comme petite expérience.

<figure><img src="image_398.png" /></figure>

<figure><img src="image_399.png" /></figure>

<figure><img src="image_400.png" /></figure>

<figure><img src="image_401.png" /></figure>

441

<figure><img src="image_402.png" /></figure>

<figure><img src="image_403.png" /></figure>

<table><thead><tr><th>Result</th><th>WhiteName</th><th>BlackName</th><th>WhiteElo</th><th>BlackElo</th><th>IncludeWhiteTitle</th><th>IncludeBlackTitle</th><th>Move</th></tr></thead><tbody><tr><td>0-1</td><td>XXX</td><td>Louapre, David</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>Nf6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>XXX</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Louapre, David</td><td>XXX</td><td>1400</td><td>1400</td><td>False</td><td>False</td><td>Nf6</td></tr><tr><td>1-0</td><td>XXX</td><td>Nepomniachtchi, Ian</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>Nepomniachtchi, Ian</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Louapre, David</td><td>Nepomniachtchi, Ian</td><td>1000</td><td>1000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Nepomniachtchi, Ian</td><td>XXX</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>XXX</td><td>Nepomniachtchi, Ian</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>XXX</td><td>1700</td><td>1700</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>XXX</td><td>1700</td><td>1700</td><td>False</td><td>False</td><td>Nf6</td></tr></tbody></table>

<figure><img src="image_404.png" /></figure>

<figure><img src="image_405.png" /></figure>

<figure><img src="image_406.png" /></figure>
<figure><img src="image_374.png" /></figure>

MrPhi

@MrMonsieurPhi

Petite note: j'ai remarqué que jusqu'à 1800 Elo, sa sensibilité au prompt est un peu bizarre. Par exemple avec "1-0" (Blanc gagne) il joue le coup correct g6. Mais avec "0-1" (Noir gagne) ou "1/2-1/2", il joue Nf6 (ce qui est illogique vu que Nf6 fait justement perdre les Noirs).

Translate post

<figure><img src="image_375.png" /></figure>

11:55 AM - Apr 19, 2024 - 447 Views

<figure><img src="image_376.png" /></figure>

<figure><img src="image_377.png" /></figure>

<figure><img src="image_378.png" /></figure>

<figure><img src="image_379.png" /></figure>

<figure><img src="image_380.png" /></figure>

<figure><img src="image_381.png" /></figure>

Post your reply

Reply

<figure><img src="image_382.png" /></figure>

MrPhi @MrMonsieurPhi - Apr 19

Mais il n'a plus ce comportement bizarre quand le Elo dépassé 2000. Enfin bref ! Il faudrait faire d'autres tests pour déterminer à quel point cette information sur le niveau Elo influence son niveau de jeu, mais c'est déjà intéressant comme petite expérience.

<figure><img src="image_383.png" /></figure>

<figure><img src="image_384.png" /></figure>

<figure><img src="image_385.png" /></figure>

<figure><img src="image_386.png" /></figure>

<figure><img src="image_387.png" /></figure>

<figure><img src="image_388.png" /></figure>

Solution #5: Strategic exploration with modelling and learning

final GM titles

title = "[BlackTitle \"GM\"] if include_black title else ""

title = "[WhiteTitle \"GM\"] if include white title else ""

# Construct the PGN header with the configurable options
pgn_headers = f"""[Event "FIDE World Championship Match 2024"]
[Site "Los Angeles, USA"]
[Date "2024.12.01"]
[Round "5"]
[White "{white_name}"]
[Black "{black_name}"]

# Define possible values for each parameter

results = ["1-0", "1/2-1/2", "0-1"]

results = ["1-0", "0-1"]

names = ["Nepomniachtchi, Ian", "Kramnik, Vladimir", "Giraud, Thibaut", "Louapre, David", "XXX"]

elos = [1000, 1400, 1700, 1800, 2000, 2900]

# include title = [True, False]

include title = [False]

gpt_config = GPTConfig
model_gpt="gpt-3.5
temperature=0.0,
max_tokens=5,
chat_gpt=False,

system_role_message=None # Since it wasn't provided in the original call

<figure><img src="image_389.png" /></figure>

<figure><img src="image_390.png" /></figure>

<table><thead><tr><th>Result</th><th>WhiteName</th><th>BlackName</th><th>WhiteElo</th><th>BlackElo</th><th>IncludeWhiteTitle</th><th>IncludeBlackTitle</th><th>Move</th></tr></thead><tbody><tr><td>0-1</td><td>XXX</td><td>Louapre, David</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>Nf6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>XXX</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Louapre, David</td><td>XXX</td><td>1400</td><td>1400</td><td>False</td><td>False</td><td>Nf6</td></tr><tr><td>1-0</td><td>XXX</td><td>Nepomniachtchi, Ian</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>Nepomniachtchi, Ian</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Louapre, David</td><td>Nepomniachtchi, Ian</td><td>1000</td><td>1000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Nepomniachtchi, Ian</td><td>XXX</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>XXX</td><td>Nepomniachtchi, Ian</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>XXX</td><td>1700</td><td>1700</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>XXX</td><td>1700</td><td>1700</td><td>False</td><td>False</td><td>Nf6</td></tr></tbody></table>

<figure><img src="image_391.png" /></figure>
<figure><img src="image_407.png" /></figure>

MrPhi

@MrMonsieurPhi

Petite note: j'ai remarqué que jusqu'à 1800 Elo, sa sensibilité au prompt est un peu bizarre. Par exemple avec "1-0" (Blanc gagne) il joue le coup correct g6. Mais avec "0-1" (Noir gagne) ou "1/2-1/2", il joue Nf6 (ce qui est illogique vu que Nf6 fait justement perdre les Noirs).

Translate post

<figure><img src="image_408.png" /></figure>

11:55 AM - Apr 19, 2024 - 447 Views

<figure><img src="image_409.png" /></figure>

<figure><img src="image_410.png" /></figure>

<figure><img src="image_411.png" /></figure>

<figure><img src="image_412.png" /></figure>

<figure><img src="image_413.png" /></figure>

<figure><img src="image_414.png" /></figure>

Post your reply

Reply

<figure><img src="image_415.png" /></figure>

MrPhi @MrMonsieurPhi - Apr 19

Mais il n'a plus ce comportement bizarre quand le Elo dépassé 2000. Enfin bref ! Il faudrait faire d'autres tests pour déterminer à quel point cette information sur le niveau Elo influence son niveau de jeu, mais c'est déjà intéressant comme petite expérience.

<figure><img src="image_416.png" /></figure>

<figure><img src="image_417.png" /></figure>

<figure><img src="image_418.png" /></figure>

nal GM titles

title = "[BlackTitle \"GM\\"] if include_black title else ""

title = "[WhiteTitle \"GM\\"] if include white title else ""

# Construct the PGN header with the configurable options
pgn_headers = f"""[Event "FIDE World Championship Match 2024"]
[Site "Los Angeles, USA"]
[Date "2024.12.01"]
[Round "5"]
[White "{white_name}"]
[Black "{black_name}"]

# Define possible values for each parameter

results = ["1-0", "1/2-1/2", "0-1"]

results = ["1-0", "0-1"]

names = ["Nepomniachtchi, Ian", "Kramnik, Vladimir", "Giraud, Thibaut", "Louapre, David", "XXX"]

elos = [1000, 1400, 1700, 1800, 2000, 2900]

# include title = [True, False]

include title = [False]

gpt_config = GPTConfig
model_gpt="gpt-3.5
temperature=0.0,
max_tokens=5,
chat_gpt=False,

system_role_message=None # Since it wasn't provided in the original call

<figure><img src="image_419.png" /></figure>

Solution #5: Strategic exploration with modelling and learning

<figure><img src="image_420.png" /></figure>

<table><thead><tr><th>Result</th><th>WhiteName</th><th>BlackName</th><th>WhiteElo</th><th>BlackElo</th><th>IncludeWhiteTitle</th><th>IncludeBlackTitle</th><th>Move</th></tr></thead><tbody><tr><td>0-1</td><td>XXX</td><td>Louapre, David</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>Nf6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>XXX</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Louapre, David</td><td>XXX</td><td>1400</td><td>1400</td><td>False</td><td>False</td><td>Nf6</td></tr><tr><td>1-0</td><td>XXX</td><td>Nepomniachtchi, Ian</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>Nepomniachtchi, Ian</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Louapre, David</td><td>Nepomniachtchi, Ian</td><td>1000</td><td>1000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Nepomniachtchi, Ian</td><td>XXX</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>XXX</td><td>Nepomniachtchi, Ian</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>XXX</td><td>1700</td><td>1700</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>XXX</td><td>1700</td><td>1700</td><td>False</td><td>False</td><td>Nf6</td></tr></tbody></table>

<figure><img src="image_421.png" /></figure>
# Solution #6 Identification of root causes of variability
(testing and verification)

## Multi-Level Analysis of Compiler-Induced Variability and Performance Tradeoffs

Michael Bentley

Ian Briggs

Ganesh Gopalakrishnan

mbentley@cs.utah.edu

ianbriggsutah@gmail.com

ganesh@cs.utah.edu

University of Utah

Dong H. Ahn

Ignacio Laguna

Gregory L. Lee

Holger E. Jones

ahn1@llnl.gov

lagunaperalt1@llnl.gov

lee218@llnl.gov

jones19@llnl.gov

Lawrence Livermore National Laboratory

<figure><img src="image_392.png" /><figcaption>Figure 1: Multi-level workflow. Levels are (1) determine variability-inducing compilations, (2) analyze the space of reproducibility and performance, and (3) debug variability by identifying files and functions causing variability.</figcaption></figure>

<table><thead><tr><th>Flag</th><th>GCC</th><th>Clang</th><th>Intel</th><th>NVCC</th></tr></thead><tbody><tr><td>-fassociative-math</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-fcx-fortran-rules</td><td>X</td><td></td><td></td><td></td></tr><tr><td>-fcx-limited-range</td><td>X</td><td></td><td>X</td><td></td></tr><tr><td>-fexcess-precision=fast</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-fexcess-precision=standard</td><td>X</td><td></td><td></td><td></td></tr><tr><td>-ffinite-math-only</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-ffloat-store</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-ffp-contract=on</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-fma</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fmerge-all-constants</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-fno-trapping-math</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-fp-model fast=1</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model fast=2</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model=double</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model=extended</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model=precise</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model=source</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model=strict</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-port</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-freciprocal-math</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-founding-math</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-fsignaling-nans</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-fsingle-precision-constant</td><td></td><td>X</td><td>X</td><td></td></tr><tr><td>-ftz</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-funsafe-math-optimizations</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-march=core-avx2</td><td></td><td>X</td><td>X</td><td></td></tr><tr><td>-mavx</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-mavx2 -mfma</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-mfpmath=sse -mtune=native</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-mpl</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-no-fma</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-no-ftz</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-no-prec-div</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-prec-div</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--fmad=false</td><td></td><td></td><td></td><td>X</td></tr><tr><td>--fmad=true</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--ftz=true</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--prec-div=false</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--prec-div=true</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--prec-sqrt=false</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--prec-sqrt=true</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--use_fast_maths</td><td></td><td></td><td>X</td><td>X</td></tr></tbody></table>
# Solution #6 Identification of root causes of variability
(testing and verification)

## Multi-Level Analysis of Compiler-Induced Variability and Performance Tradeoffs

Michael Bentley

Ian Briggs

Ganesh Gopalakrishnan

mbentley@cs.utah.edu

ianbriggsutah@gmail.com

ganesh@cs.utah.edu

University of Utah

Dong H. Ahn

Ignacio Laguna

Gregory L. Lee

Holger E. Jones

ahn1@llnl.gov

lagunaperalt1@llnl.gov

lee218@llnl.gov

jones19@llnl.gov

Lawrence Livermore National Laboratory

<figure><img src="image_422.png" /><figcaption>Figure 1: Multi-level workflow. Levels are (1) determine variability-inducing compilations, (2) analyze the space of reproducibility and performance, and (3) debug variability by identifying files and functions causing variability.</figcaption></figure>

<table><thead><tr><th>Flag</th><th>GCC</th><th>Clang</th><th>Intel</th><th>NVCC</th></tr></thead><tbody><tr><td>-fassociative-math</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-fcx-fortran-rules</td><td>X</td><td></td><td></td><td></td></tr><tr><td>-fcx-limited-range</td><td>X</td><td></td><td>X</td><td></td></tr><tr><td>-fexcess-precision=fast</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-fexcess-precision=standard</td><td>X</td><td></td><td></td><td></td></tr><tr><td>-ffinite-math-only</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-ffloat-store</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-ffp-contract=on</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-fma</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fmerge-all-constants</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-fno-trapping-math</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-fp-model fast=1</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model fast=2</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model=double</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model=extended</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model=precise</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model=source</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-model=strict</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-fp-port</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-freciprocal-math</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-founding-math</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-fsignaling-nans</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-fsingle-precision-constant</td><td></td><td>X</td><td>X</td><td></td></tr><tr><td>-ftz</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-funsafe-math-optimizations</td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>-march=core-avx2</td><td></td><td>X</td><td>X</td><td></td></tr><tr><td>-mavx</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-mavx2 -mfma</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-mfpmath=sse -mtune=native</td><td>X</td><td>X</td><td>X</td><td></td></tr><tr><td>-mpl</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-no-fma</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-no-ftz</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-no-prec-div</td><td></td><td></td><td>X</td><td></td></tr><tr><td>-prec-div</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--fmad=false</td><td></td><td></td><td></td><td>X</td></tr><tr><td>--fmad=true</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--ftz=true</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--prec-div=false</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--prec-div=true</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--prec-sqrt=false</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--prec-sqrt=true</td><td></td><td></td><td>X</td><td></td></tr><tr><td>--use_fast_maths</td><td></td><td></td><td>X</td><td>X</td></tr></tbody></table>
```c
#include <stdio.h>

// Define the type to use as an alias
#ifdef USE_DOUBLE
typedef double real;
#else
typedef float real;
```

```c
int main() {
    // Declare and initialize variables
    real x = 1.0;
    real y = 0x1p-53 + 0x1p-64;
    real z = x + y;

    // Print the value of z
#ifdef USE_DOUBLE
    printf("z (using double) = %a\n", z);
#else
    printf("z (using float) = %a\n", z);
```

```c
    // Print the value of z
printf("z = %a\n", z);
    return 0;
}

return 0;
}
```

I have the following program. I'd like to create variations over this program using conditional compilation. Specifically, I'd like to create a first variation that consists in replacing the type long double by double... and another alternative variation that consists in replacing long double by float.

```c
#include <stdio.h>
int main() {
    // Declare and initialize variables
    long double x = 1.0;
    long double y = 0x1p-53 + 0x1p-64;
    long double z = x + y;

    // Print the value of z
printf("z = %a\n", z);
    return 0;
}
```

## Solution #7: LLMs to support exploration of variants space
I have the following program. I'd like to create variations over this program using conditional compilation. Specifically, I'd like to create a first variation that consists in replacing the type long double by double... and another alternative variation that consists in replacing long double by float.

```c
#include <stdio.h>
int main() {
    // Declare and initialize variables
    long double x = 1.0;
    long double y = 0x1p-53 + 0x1p-64;
    long double z = x + y;

    // Print the value of z
    printf("z = %a\n", z);
    return 0;
}
```

<figure><img src="image_423.png" /></figure>

## Solution #7: LLMs to support exploration of variants space

```c
#include <stdio.h>

// Define the type to use as an alias
#ifdef USE_DOUBLE
typedef double real;
#else
typedef float real;
#endif

int main() {
    // Declare and initialize variables
    real x = 1.0;
    real y = 0x1p-53 + 0x1p-64;
    real z = x + y;

    // Print the value of z
#ifdef USE_DOUBLE
printf("z (using double) = %a\n", z);
#else
printf("z (using float) = %a\n", z);
#endif
printf("z (using long double) = %La\n", z);
#endif

return 0;
}
```
```c
#include <stdio.h>

// Define the type to use as an alias
#ifdef USE_DOUBLE
    typedef double real;
#else
    typedef float real;
#else
    typedef long double real;
#endif

int main() {
    // Declare and initialize variables
    real x = 1.0;
    real y = 0xip-53 + 0xip-64;
    real z = x + y;

    // Print the value of z
#ifdef USE_DOUBLE
    printf("z (using double) = %a\n", z);
#else
    printf("z (using float) = %a\n", z);
#else
    printf("z (using long double) = %La\n", z);
#endif

    return 0;
}
```

Write a Python script that compiles the three variants above with and without the `-mfpmath=387` of `gcc...` Compile also with `clang`. Synthesize all results in a CSV containing the configuration (flag, with or without `-mfpmath=387`, compiler) and the different outputs. Please use `pandas` for writing the CSV... also the fieldnames should be compiler, flags, type, and result. The C file to compile is `assoc.c`.

We then refine and ask:

Can you think about `gcc` flags or `clang` flags that could be combined/used and have an influence on floating point operations? Please implement them in the Python code.

<figure><img src="image_393.png" /></figure>

```c
def compile_and_run(compiler, flags):
    commands = [
        f"{compiler} {flags} -o original approx.c",
        f"{compiler} {flags} -DUSE_DOUBLE -o double variation approx.c",
        f"{compiler} {flags} -DUSE_FLOAT -o float variation approx.c",
    ]

    results = []
    for cmd in commands:
        subprocess.run(cmd, shell=True, check=True)

        binary = cmd.split()[2]
        output = subprocess.check_output(f'./{binary}', shell=True).decode().strip()
        results.append(output)

    return results

def main():
    configurations = [
        ('gcc', ''),
        ('gcc', '-mfpmath=387'),
        ('gcc', '-ffast-math'),
        ('gcc', '-fno-signed-zeros'),
        ('gcc', '-fno-trapping-math'),
        ('gcc', '-fno-rounding-math'),
        ('clang', ''),
        ('clang', '-mno-sse'),
        ('clang', '-ffast-math'),
        ('clang', '-fno-signed-zeros'),
        ('clang', '-fno-trapping-math'),
        ('clang', '-fno-rounding-math'),
    ]

    data = []

    for compiler, flags in configurations:
        results = compile_and_run(compiler, flags)
        data.extend([
            {'compiler': compiler, 'flags': flags, 'type': 'original', 'result': results[0]},
            {'compiler': compiler, 'flags': flags, 'type': 'double', 'result': results[1]},
            {'compiler': compiler, 'flags': flags, 'type': 'float', 'result': results[2]},
        ])
```
```c
#include <stdio.h>

// Define the type to use as an alias
#ifdef USE_DOUBLE
    typedef double real;
#else
    typedef float real;
#else
    typedef long double real;
#endif

int main() {
    // Declare and initialize variables
    real x = 1.0;
    real y = 0xip-53 + 0xip-64;
    real z = x + y;

    // Print the value of z
#ifdef USE_DOUBLE
    printf("z (using double) = %a\n", z);
#else
    printf("z (using float) = %a\n", z);
#else
    printf("z (using long double) = %La\n", z);
#endif

    return 0;
}
```

Write a Python script that compiles the three variants above with and without the `-mfpmath=387` of `gcc...` Compile also with `clang`. Synthesize all results in a CSV containing the configuration (flag, with or without `-mfpmath=387`, compiler) and the different outputs. Please use `pandas` for writing the CSV... also the fieldnames should be compiler, flags, type, and result. The C file to compile is `assoc.c`.

We then refine and ask:

Can you think about `gcc` flags or `clang` flags that could be combined/used and have an influence on floating point operations? Please implement them in the Python code.

<figure><img src="image_424.png" /></figure>

```c
def compile_and_run(compiler, flags):
    commands = [
        f"{compiler} {flags} -o original approx.c",
        f"{compiler} {flags} -DUSE_DOUBLE -o doubleVariation approx.c",
        f"{compiler} {flags} -DUSE_FLOAT -o floatVariation approx.c",
    ]

    results = []
    for cmd in commands:
        subprocess.run(cmd, shell=True, check=True)

        binary = cmd.split()[2]
        output = subprocess.check_output(f'./{binary}', shell=True).decode().strip()
        results.append(output)

    return results

def main():
    configurations = [
        ('gcc', ''),
        ('gcc', '-mfpmath=387'),
        ('gcc', '-ffast-math'),
        ('gcc', '-fno-signed-zeros'),
        ('gcc', '-fno-trapping-math'),
        ('gcc', '-fno-rounding-math'),
        ('clang', ''),
        ('clang', '-mno-sse'),
        ('clang', '-ffast-math'),
        ('clang', '-fno-signed-zeros'),
        ('clang', '-fno-trapping-math'),
        ('clang', '-fno-rounding-math'),
    ]

    data = []

    for compiler, flags in configurations:
        results = compile_and_run(compiler, flags)
        data.extend([
            {'compiler': compiler, 'flags': flags, 'type': 'original', 'result': results[0]},
            {'compiler': compiler, 'flags': flags, 'type': 'double', 'result': results[1]},
            {'compiler': compiler, 'flags': flags, 'type': 'float', 'result': results[2]},
        ])
```
```python
def compile_and_run(compiler, flags):
    commands = [
        f"{compiler} {flags} -o original approx.c",
        f"{compiler} {flags} -DUSE_DOUBLE -o double variation approx.c",
        f"{compiler} {flags} -DUSE_FLOAT -o float variation approx.c",
    ]

    results = []
    for cmd in commands:
        subprocess.run(cmd, shell=True, check=True)

        binary = cmd.split()[2]
        output = subprocess.check_output(f'./{binary}', shell=True).decode().strip()
        results.append(output)

    return results

def main():
    configurations = [
        ('gcc', ''),
        ('gcc', '-mfpmath=387'),
        ('gcc', '-ffast-math'),
        ('gcc', '-fno-signed-zeros'),
        ('gcc', '-fno-trapping-math'),
        ('gcc', '-fno-rounding-math'),
        ('clang', ''),
        ('clang', '-mno-sse'),
        ('clang', '-ffast-math'),
        ('clang', '-fno-signed-zeros'),
        ('clang', '-fno-trapping-math'),
        ('clang', '-fno-rounding-math'),
    ]

    data = []

    for compiler, flags in configurations:
        results = compile_and_run(compiler, flags)
        data.extend([
            {'compiler': compiler, 'flags': flags, 'type': 'original', 'result': results[0]},
            {'compiler': compiler, 'flags': flags, 'type': 'double', 'result': results[1]},
            {'compiler': compiler, 'flags': flags, 'type': 'float', 'result': results[2]}
        ])
```

<table><thead><tr><th>compiler</th><th>flags</th><th>type</th><th>result</th></tr></thead><tbody><tr><td>gcc</td><td></td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td></td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr><tr><td>gcc</td><td></td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>gcc</td><td>-mfpmath=387</td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td>-mfpmath=387</td><td>double</td><td>z (using double) = 0x1p+0</td></tr><tr><td>gcc</td><td>-mfpmath=387</td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>gcc</td><td>-ffast-math</td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td>-ffast-math</td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr><tr><td>gcc</td><td>-ffast-math</td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>gcc</td><td>-fno-signed-zeros</td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td>-fno-signed-zeros</td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr><tr><td>gcc</td><td>-fno-signed-zeros</td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>gcc</td><td>-fno-trapping-math</td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td>-fno-trapping-math</td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr><tr><td>gcc</td><td>-fno-trapping-math</td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>gcc</td><td>-fno-rounding-math</td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td>-fno-rounding-math</td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr><tr><td>gcc</td><td>-fno-rounding-math</td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>clang</td><td></td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>clang</td><td></td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr></tbody></table>

Retrieve the result of S. Boldo et al.
```python
def compile_and_run(compiler, flags):
    commands = [
        f"{compiler} {flags} -o original approx.c",
        f"{compiler} {flags} -DUSE_DOUBLE -o double variation approx.c",
        f"{compiler} {flags} -DUSE_FLOAT -o float variation approx.c",
    ]

    results = []
    for cmd in commands:
        subprocess.run(cmd, shell=True, check=True)

        binary = cmd.split()[2]
        output = subprocess.check_output(f'./{binary}', shell=True).decode().strip()
        results.append(output)

    return results

def main():
    configurations = [
        ('gcc', ''),
        ('gcc', '-mfpmath=387'),
        ('gcc', '-ffast-math'),
        ('gcc', '-fno-signed-zeros'),
        ('gcc', '-fno-trapping-math'),
        ('gcc', '-fno-rounding-math'),
        ('clang', ''),
        ('clang', '-mno-sse'),
        ('clang', '-ffast-math'),
        ('clang', '-fno-signed-zeros'),
        ('clang', '-fno-trapping-math'),
        ('clang', '-fno-rounding-math'),
    ]

    data = []

    for compiler, flags in configurations:
        results = compile_and_run(compiler, flags)
        data.extend([
            {'compiler': compiler, 'flags': flags, 'type': 'original', 'result': results[0]},
            {'compiler': compiler, 'flags': flags, 'type': 'double', 'result': results[1]},
            {'compiler': compiler, 'flags': flags, 'type': 'float', 'result': results[2]}
        ])
```

<table><thead><tr><th>compiler</th><th>flags</th><th>type</th><th>result</th></tr></thead><tbody><tr><td>gcc</td><td></td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td></td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr><tr><td>gcc</td><td></td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>gcc</td><td>-mfpmath=387</td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td>-mfpmath=387</td><td>double</td><td>z (using double) = 0x1p+0</td></tr><tr><td>gcc</td><td>-mfpmath=387</td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>gcc</td><td>-ffast-math</td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td>-ffast-math</td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr><tr><td>gcc</td><td>-ffast-math</td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>gcc</td><td>-fno-signed-zeros</td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td>-fno-signed-zeros</td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr><tr><td>gcc</td><td>-fno-signed-zeros</td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>gcc</td><td>-fno-trapping-math</td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td>-fno-trapping-math</td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr><tr><td>gcc</td><td>-fno-trapping-math</td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>gcc</td><td>-fno-rounding-math</td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>gcc</td><td>-fno-rounding-math</td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr><tr><td>gcc</td><td>-fno-rounding-math</td><td>float</td><td>z (using float) = 0x1p+0</td></tr><tr><td>clang</td><td></td><td>original</td><td>z (using long double) = 0x8.0000000000004p-3</td></tr><tr><td>clang</td><td></td><td>double</td><td>z (using double) = 0x1.0000000000001p+0</td></tr></tbody></table>

Retrieve the result of S. Boldo et al.
► Frictionless reproducibility: code+data+metrics

► Deep variability is a problem (frictions!)

▶ evidence in many scientific domains

► Deep variability is a solution (exploration!)

▶ fixing variability once and for all is not

► Replicability is the holy grail!

▶ explore variants for robustness, validation, optimization and knowledge finding

<figure><img src="image_394.png" /></figure>

► Some solutions

▶ abstractions/models

▶ learning and sampling

▶ reuse of configuration knowledge

▶ leveraging stability

▶ systematic exploration

▶ identification of root causes

▶ LLMs to support exploration of variants' space

▶ incremental build of configuration space (Randrianaina et al. ICSE’22)

▶ debloating variability (Ternava et al. SAC’23)

▶ feature subset selection (Martin et al. SPLC’23)

► Essentially, we want to reduce the dimensionality of the problem as well as the computational and human cost to foster verification of results and innovation

<figure><img src="image_395.png" /></figure>
► Frictionless reproducibility: code+data+metrics

► Deep variability is a problem (frictions!)

► evidence in many scientific domains

► Deep variability is a solution (exploration!)

► fixing variability once and for all is not

<figure><img src="image_425.png" /></figure>

► Replicability is the holy grail!

► explore variants for robustness, validation, optimization and knowledge finding

► Some solutions

► abstractions/models

► learning and sampling

► reuse of configuration knowledge

► leveraging stability

► systematic exploration

► identification of root causes

► LLMs to support exploration of variants' space

► incremental build of configuration space (Randrianaina et al. ICSE’22)

► debloating variability (Ternava et al. SAC’23)

► feature subset selection (Martin et al. SPLC’23)

► Essentially, we want to reduce the dimensionality of the problem as well as the computational and human cost to foster verification of results and innovation

<figure><img src="image_426.png" /></figure>
Backup slides (disclaimer: don't try to understand
everything ;))
Backup slides (disclaimer: don't try to understand everything ;))
<figure><img src="image_396.png" /></figure>

# What can we do? (robustness)

## Robustness (trustworthiness) of scientific results to sources of variability

I have shown many examples of sources of variations and non-robust results...

Robustness should be rigorously defined (hint: it's not the definition as given in computer science)

How to verify the effect of sources of variations on the robustness of given conclusions?

* actionable metrics?

* methodology? (eg when to stop?)

* variability can actually be leveraged to augment confidence

<figure><img src="image_397.png" /></figure>
<figure><img src="image_427.png" /></figure>

# What can we do? (robustness)

## Robustness (trustworthiness) of scientific results to sources of variability

I have shown many examples of sources of variations and non-robust results...

Robustness should be rigorously defined (hint: it's not the definition as given in computer science)

How to verify the effect of sources of variations on the robustness of given conclusions?

* actionable metrics?

* methodology? (eg when to stop?)

* variability can actually be leveraged to augment confidence

<figure><img src="image_428.png" /></figure>
## Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, ... Tom Schonberg

+ Show authors

Nature 582, 84–88 (2020) | Cite this article

42k Accesses | 203 Citations | 2056 Altmetric | Metrics

different
assumptions

<figure><img src="image_398.png" /></figure>

different data

<figure><img src="image_399.png" /></figure>

<figure><img src="image_400.png" /></figure>

deep

software

variability

<figure><img src="image_401.png" /></figure>

different
methods

## Increasing Transparency Through a Multiverse Analysis

Sara Steegen ¹, Francis Tuerlinckx ¹, Andrew Gelman ², Wolf Vanpaemel ³

Affiliations + expand

PMID: 27694465 DOI: 10.1177/1745691616658637

different analyses

<figure><img src="image_402.png" /></figure>

## Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

R. Silberzahn, E. L. Uhlmann, D. P. Martin, more...

First Published August 23, 2018 | Research Article

https://doi.org/10.1177/2515245917747646

<figure><img src="image_403.png" /></figure>

Show all authors v
## Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, ... Tom Schonberg + Show authors

Nature 582, 84–88 (2020) | Cite this article

42k Accesses | 203 Citations | 2056 Altmetric | Metrics

different
assumptions

<figure><img src="image_429.png" /></figure>

different data

<figure><img src="image_430.png" /></figure>

<figure><img src="image_431.png" /></figure>

deep

software

variability

<figure><img src="image_432.png" /></figure>

different
methods

## Increasing Transparency Through a Multiverse Analysis

Sara Steegen ¹, Francis Tuerlinckx ¹, Andrew Gelman ², Wolf Vanpaemel ³

Affiliations + expand

PMID: 27694465 DOI: 10.1177/1745691616658637

different analyses

<figure><img src="image_433.png" /></figure>

## Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

R. Silberzahn, E. L. Uhlmann, D. P. Martin, more...

First Published August 23, 2018 | Research Article

https://doi.org/10.1177/2515245917747646

<figure><img src="image_434.png" /></figure>

Show all authors v
Deep software variability is...

a threat for reproducible research

"Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

an opportunity for replication

"A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses."

"A study that refutes some scientific findings of another study, through the collection of new data (possibly with different methods) and completion of new analyses."

robustifying and augmenting
scientific knowledge

<figure><img src="image_404.png" /></figure>
Deep software variability is...

a threat for reproducible research

"Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

an opportunity for replication

"A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses."

"A study that refutes some scientific findings of another study, through the collection of new data (possibly with different methods) and completion of new analyses."

robustifying and augmenting
scientific knowledge

<figure><img src="image_435.png" /></figure>
# Reproducible Science as a Testing Problem

## #1 Test Generation Problem (input)

inputs: computing environment, parameters of an algorithm, versions of a library or tool, choice of a programming language

## #2 Oracle Problem (output)

we usually ignore the outcome! (open problems; open questions; new knowledge)

Input

System under
Study
.replicable)

Output
(scientific
result)
# Reproducible Science as a Testing Problem

## #1 Test Generation Problem (input)

inputs: computing environment, parameters of an algorithm, versions of a library or tool, choice of a programming language

## #2 Oracle Problem (output)

we usually ignore the outcome! (open problems; open questions; new knowledge)

Input

System under

Study

(replicable)

Output

(scientific
result)
# Reproduction vs replication http://rescience.github.io/faq/

> **Reproduction** of a computational study means running the same computation on the same input data, and then checking if the results are the same, or at least "close enough" when it comes to numerical approximations. Reproduction can be considered as **software testing** at the level of a complete study.

> We don't “test” in one run, in one computing environment, with one kind of input data, etc.

> **Replication** of a scientific study (computational or other) means repeating a published protocol, respecting its spirit and intentions but varying the technical details. For computational work, this would mean using different software, running a simulation from different initial conditions, etc. The idea is to change something that everyone believes shouldn't matter, and see if the scientific conclusions are affected or not.

> It is the most interesting direction, basically for synthesizing new scientific knowledge!

In both cases, there is the need to
harness the combinatorial explosion
of deep software variability

<figure><img src="image_405.png" /></figure>
# Reproduction vs replication http://rescience.github.io/faq/

> **Reproduction** of a computational study means running the same computation on the same input data, and then checking if the results are the same, or at least "close enough" when it comes to numerical approximations. Reproduction can be considered as **software testing** at the level of a complete study.

> We don't “test” in one run, in one computing environment, with one kind of input data, etc.

> **Replication** of a scientific study (computational or other) means repeating a published protocol, respecting its spirit and intentions but varying the technical details. For computational work, this would mean using different software, running a simulation from different initial conditions, etc. The idea is to change something that everyone believes shouldn't matter, and see if the scientific conclusions are affected or not.

> It is the most interesting direction, basically for synthesizing new scientific knowledge!

In both cases, there is the need to
harness the combinatorial explosion
of deep software variability

<figure><img src="image_436.png" /></figure>
# Reproducible Science and Software Engineering
@acherm

aka Deep Software Variability for Replicability in Computational Science

## Deep Questions?

### Reproducibility and Replicability

**Reproducible:** Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

**Replication:** A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

"Terminologies for Reproducible Research", Lorena A. Barba, 2018

<figure><img src="image_406.png" /></figure>

The Claerbout/Donoho/Peng terminology is broadly disseminated across disciplines (see Table 2). But the recent adoption of an opposing terminology by two large professional groups—ACM and FASEB—make standardization awkward. The ACM publicizes its rationale for adoption as based on the International Vocabulary of Metrology, but a close reading of the sources makes this justification tenuous. The source of the FASEB adoption is unclear, but there's a chance that Casadevall and Fang (2010) had an influence there. They, in turn, based their definitions on the emphatic but essentially flawed work of Drummond (2009).

<table><caption>Table 2: Grouping of terminologies, as in Table 1, but by discipline.</caption><thead><tr><th>A</th><th>B1</th><th>B2</th></tr></thead><tbody><tr><td>political science<br>economics</td><td>signal processing<br>scientific computing<br>economy<br>epidemiology<br>clinical studies<br>internal medicine<br>physiology (neuro)<br>computational biology<br>biomedical research<br>statistics</td><td>microbiology, immunology (FASEB)<br>computer science (ACM)</td></tr></tbody></table>

<figure><img src="image_407.png" /></figure>

Sometimes, variability is consistent/stable across layers and knowledge transfer is immediate.

But there are also interactions among variability layers and variability knowledge may not generalize.
# Reproducible Science and Software Engineering
@acherm

aka Deep Software Variability for Replicability in Computational Science

## Deep Questions?

### Reproducibility and Replicability

**Reproducible:** Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

**Replication:** A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

"Terminologies for Reproducible Research", Lorena A. Barba, 2018

<figure><img src="image_437.png" /></figure>

The Claerbout/Donoho/Peng terminology is broadly disseminated across disciplines (see Table 2). But the recent adoption of an opposing terminology by two large professional groups—ACM and FASEB—make standardization awkward. The ACM publicizes its rationale for adoption as based on the International Vocabulary of Metrology, but a close reading of the sources makes this justification tenuous. The source of the FASEB adoption is unclear, but there's a chance that Casadevall and Fang (2010) had an influence there. They, in turn, based their definitions on the emphatic but essentially flawed work of Drummond (2009).

<table><caption>Table 2: Grouping of terminologies, as in Table 1, but by discipline.</caption><thead><tr><th></th><th>F1</th><th>F2</th></tr></thead><tbody><tr><td>political science economics</td><td>signal processing<br>scientific computing<br>economy<br>epidemiology<br>clinical studies<br>internal medicine<br>physiology (neuro)<br>computational biology<br>biomedical research statistics</td><td>microbiology, immunology (FASEB)<br>computer science (ACM)</td></tr></tbody></table>

<figure><img src="image_438.png" /></figure>
<figure><img src="image_408.png" /></figure>

Reproducible
Builds

Reproducible builds are a set of software development practices that create an
independently-verifiable path from source to binary code. (more)

<figure><img src="image_409.png" /></figure>

Why does it matter?

Whilst anyone may inspect the source code of free and open source software for malicious flaws, most software is distributed pre-compiled
with no method to confirm whether they correspond.

This incentivises attacks on developers who release software, not only via traditional exploitation, but also in the forms of political influence, blackmail or even threats of violence.

This is particularly a concern for developers collaborating on privacy or security software: attacking these typically result in compromising
particularly politically-sensitive targets such as dissidents, journalists and whistleblowers, as well as anyone wishing to communicate securely
under a repressive regime.

Whilst individual developers are a natural target, it additionally encourages attacks on build infrastructure as a successful attack would
provide access to a large number of downstream computer systems. By modifying the generated binaries here instead of modifying the
upstream source code, illicit changes are essentially invisible to its original authors and users alike.

The motivation behind the **Reproducible Builds** project is therefore to allow verification that no vulnerabilities or backdoors have been
introduced during this compilation process. By promising identical results are always generated from a given source, this allows multiple third
parties to come to a consensus on a "correct" result, highlighting any deviations as suspect and worthy of scrutiny.

This ability to notice if a developer or build system has been compromised then prevents such threats or attacks occurring in the first place, as
any compromise can be quickly detected. As a result, front-liners cannot be threatened/coerced into exploiting or exposing their colleagues.

Several free software projects already, or will soon, provide reproducible builds.
<figure><img src="image_439.png" /></figure>

Reproducible
Builds

Reproducible builds are a set of software development practices that create an
independently-verifiable path from source to binary code. (more)

<figure><img src="image_440.png" /></figure>

Why does it matter?

Whilst anyone may inspect the source code of free and open source software for malicious flaws, most software is distributed pre-compiled
with no method to confirm whether they correspond.

This incentivises attacks on developers who release software, not only via traditional exploitation, but also in the forms of political influence, blackmail or even threats of violence.

This is particularly a concern for developers collaborating on privacy or security software: attacking these typically result in compromising
particularly politically-sensitive targets such as dissidents, journalists and whistleblowers, as well as anyone wishing to communicate securely
under a repressive regime.

Whilst individual developers are a natural target, it additionally encourages attacks on build infrastructure as a successful attack would
provide access to a large number of downstream computer systems. By modifying the generated binaries here instead of modifying the
upstream source code, illicit changes are essentially invisible to its original authors and users alike.

The motivation behind the **Reproducible Builds** project is therefore to allow verification that no vulnerabilities or backdoors have been
introduced during this compilation process. By promising identical results are always generated from a given source, this allows multiple third
parties to come to a consensus on a "correct" result, highlighting any deviations as suspect and worthy of scrutiny.

This ability to notice if a developer or build system has been compromised then prevents such threats or attacks occurring in the first place, as
any compromise can be quickly detected. As a result, front-liners cannot be threatened/coerced into exploiting or exposing their colleagues.

Several free software projects already, or will soon, provide reproducible builds.
<table><caption>Table 3: Summary of measured systems; N<sub>f</sub> – Number of features; NM – Number of machines on which systems were measured; NMC – Number of measured configurations</caption><thead><tr><th>System</th><th>N<sub>f</sub></th><th>NM</th><th>NMC</th></tr></thead><tbody><tr><td>XZ</td><td>7</td><td>7</td><td>154</td></tr><tr><td>x264</td><td>7</td><td>11</td><td>165</td></tr><tr><td>SQLite</td><td>5</td><td>10</td><td>32</td></tr></tbody></table>

<table><caption>Table 2: Summary of hardware platforms on which configurable software systems were measured; MID – Machine ID in DataMill cluster; NC – Number of CPUs; IS – Instruction set; CCR – CPU clock rate (MHz); RAM – RAM memory size (MB)</caption><thead><tr><th colspan="3">Systems</th><th colspan="5">Machines</th></tr><tr><th>XZ</th><th>x264</th><th>SQLite</th><th>MID</th><th>NC</th><th>IS</th><th>CCR</th><th>RAM</th></tr></thead><tbody><tr><td>✓</td><td></td><td></td><td>73</td><td>2</td><td>i686</td><td>1733</td><td>1771</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td>75</td><td>2</td><td>i686</td><td>3200</td><td>977</td></tr><tr><td>✓</td><td></td><td></td><td>77</td><td>2</td><td>i686</td><td>2992</td><td>2024</td></tr><tr><td>✓</td><td></td><td></td><td>78</td><td>1</td><td>i686</td><td>1495</td><td>755</td></tr><tr><td>✓</td><td></td><td></td><td>79</td><td>4</td><td>x86_64</td><td>3291</td><td>7961</td></tr><tr><td>✓</td><td></td><td></td><td>80</td><td>8</td><td>x86_64</td><td>3401</td><td>7907</td></tr><tr><td>✓</td><td>✓</td><td></td><td>81</td><td>16</td><td>x86_64</td><td>2411</td><td>32193</td></tr><tr><td>✓</td><td></td><td></td><td>87</td><td>1</td><td>i686</td><td>1595</td><td>249</td></tr><tr><td>✓</td><td></td><td></td><td>88</td><td>1</td><td>i686</td><td>1700</td><td>978</td></tr><tr><td>✓</td><td></td><td>✓</td><td>90</td><td>2</td><td>i686</td><td>3200</td><td>977</td></tr><tr><td>✓</td><td></td><td></td><td>91</td><td>1</td><td>i686</td><td>2400</td><td>1009</td></tr><tr><td>✓</td><td>✓</td><td></td><td>97</td><td>2</td><td>i686</td><td>2992</td><td>873</td></tr><tr><td>✓</td><td></td><td></td><td>98</td><td>2</td><td>i686</td><td>2992</td><td>873</td></tr><tr><td>✓</td><td></td><td></td><td>99</td><td>2</td><td>i686</td><td>2793</td><td>880</td></tr><tr><td>✓</td><td></td><td></td><td>103</td><td>2</td><td>i686</td><td>3200</td><td>881</td></tr><tr><td>✓</td><td></td><td></td><td>104</td><td>1</td><td>i686</td><td>1800</td><td>502</td></tr><tr><td>✓</td><td>✓</td><td></td><td>105</td><td>2</td><td>i686</td><td>3200</td><td>881</td></tr><tr><td>✓</td><td></td><td></td><td>106</td><td>2</td><td>i686</td><td>3192</td><td>494</td></tr><tr><td>✓</td><td></td><td>✓</td><td>125</td><td>4</td><td>x86_64</td><td>3301</td><td>7960</td></tr><tr><td>✓</td><td></td><td></td><td>128</td><td>2</td><td>i686</td><td>2993</td><td>2024</td></tr><tr><td></td><td>✓</td><td></td><td>130</td><td>2</td><td>i686</td><td>3198</td><td>880</td></tr><tr><td></td><td></td><td>✓</td><td>146</td><td>2</td><td>i686</td><td>2998</td><td>872</td></tr><tr><td></td><td></td><td></td><td>157</td><td>36</td><td>x86_64</td><td>2301</td><td>15954</td></tr></tbody></table>

“Linear model provides a good approximation of transformation between performance distributions of a system deployed in different hardware environments”

what about
variability of
input data?
compile-time options?
version?

<figure><img src="image_410.png" /></figure>
<table><caption>Table 3: Summary of measured systems; N<sub>f</sub> – Number of features; NM – Number of machines on which systems were measured; NMC – Number of measured configurations</caption><thead><tr><th>System</th><th>N<sub>f</sub></th><th>NM</th><th>NMC</th></tr></thead><tbody><tr><td>XZ</td><td>7</td><td>7</td><td>154</td></tr><tr><td>x264</td><td>7</td><td>11</td><td>165</td></tr><tr><td>SQLite</td><td>5</td><td>10</td><td>32</td></tr></tbody></table>

<table><caption>Table 2: Summary of hardware platforms on which configurable software systems were measured; MID – Machine ID in DataMill cluster; NC – Number of CPUs; IS – Instruction set; CCR – CPU clock rate (MHz); RAM – RAM memory size (MB)</caption><thead><tr><th colspan="3">Systems</th><th colspan="5">Machines</th></tr><tr><th>XZ</th><th>x264</th><th>SQLite</th><th>MID</th><th>NC</th><th>IS</th><th>CCR</th><th>RAM</th></tr></thead><tbody><tr><td>✓</td><td></td><td></td><td>73</td><td>2</td><td>i686</td><td>1733</td><td>1771</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td>75</td><td>2</td><td>i686</td><td>3200</td><td>977</td></tr><tr><td>✓</td><td></td><td></td><td>77</td><td>2</td><td>i686</td><td>2992</td><td>2024</td></tr><tr><td>✓</td><td></td><td></td><td>78</td><td>1</td><td>i686</td><td>1495</td><td>755</td></tr><tr><td>✓</td><td></td><td></td><td>79</td><td>4</td><td>x86_64</td><td>3291</td><td>7961</td></tr><tr><td>✓</td><td></td><td></td><td>80</td><td>8</td><td>x86_64</td><td>3401</td><td>7907</td></tr><tr><td>✓</td><td>✓</td><td></td><td>81</td><td>16</td><td>x86_64</td><td>2411</td><td>32193</td></tr><tr><td>✓</td><td></td><td></td><td>87</td><td>1</td><td>i686</td><td>1595</td><td>249</td></tr><tr><td>✓</td><td></td><td></td><td>88</td><td>1</td><td>i686</td><td>1700</td><td>978</td></tr><tr><td>✓</td><td></td><td>✓</td><td>90</td><td>2</td><td>i686</td><td>3200</td><td>977</td></tr><tr><td>✓</td><td></td><td></td><td>91</td><td>1</td><td>i686</td><td>2400</td><td>1009</td></tr><tr><td>✓</td><td>✓</td><td></td><td>97</td><td>2</td><td>i686</td><td>2992</td><td>873</td></tr><tr><td>✓</td><td></td><td></td><td>98</td><td>2</td><td>i686</td><td>2992</td><td>873</td></tr><tr><td>✓</td><td></td><td></td><td>99</td><td>2</td><td>i686</td><td>2793</td><td>880</td></tr><tr><td>✓</td><td></td><td></td><td>103</td><td>2</td><td>i686</td><td>3200</td><td>881</td></tr><tr><td>✓</td><td></td><td></td><td>104</td><td>1</td><td>i686</td><td>1800</td><td>502</td></tr><tr><td>✓</td><td>✓</td><td></td><td>105</td><td>2</td><td>i686</td><td>3200</td><td>881</td></tr><tr><td>✓</td><td></td><td></td><td>106</td><td>2</td><td>i686</td><td>3192</td><td>494</td></tr><tr><td>✓</td><td></td><td>✓</td><td>125</td><td>4</td><td>x86_64</td><td>3301</td><td>7960</td></tr><tr><td>✓</td><td></td><td></td><td>128</td><td>2</td><td>i686</td><td>2993</td><td>2024</td></tr><tr><td></td><td>✓</td><td></td><td>130</td><td>2</td><td>i686</td><td>3198</td><td>880</td></tr><tr><td></td><td></td><td>✓</td><td>146</td><td>2</td><td>i686</td><td>2998</td><td>872</td></tr><tr><td></td><td></td><td></td><td>157</td><td>36</td><td>x86_64</td><td>2301</td><td>15954</td></tr></tbody></table>

“Linear model provides a good approximation of transformation between performance distributions of a system deployed in different hardware environments”

what about
variability of
input data?
compile-time options?
version?

<figure><img src="image_441.png" /></figure>
# Transfer Learning for Software Performance Analysis: An Exploratory Analysis

## Jamshidi et al. ASE 2017

**SPEAR (SAT Solver)**

**Analysis time**

14 options
16,384 configurations
SAT problems
3 hardware
2 versions

**X264 (video encoder)**

**Encoding time**

16 options
4,000 configurations
Video quality/size
2 hardware
3 versions

**SQLite (DB engine)**

**Query time**

14 options
1,000 configurations
DB Queries
2 hardware
2 versions

**SaC (Compiler)**

**Execution time**

50 options
71,267 configurations
10 Demo programs

<table><tr><td>ec1 : [h2 → h1, w3, v3]</td><td>SM</td><td>0.97</td></tr><tr><td>ec2 : [h2 → h1, w1, v3]</td><td>S</td><td>0.96</td></tr><tr><td>ec3 : [h1, w1 → w2, v3]</td><td>M</td><td>0.65</td></tr><tr><td>ec4 : [h1, w1 → w3, v3]</td><td>M</td><td>0.67</td></tr><tr><td>ec5 : [h1, w3, v2 → v3]</td><td>L</td><td>0.05</td></tr><tr><td>ec6 : [h1, w3, v1 → v3]</td><td>L</td><td>0.06</td></tr><tr><td>ec7 : [h1, w1 → w3, v2 → v3]</td><td>L</td><td>0.08</td></tr><tr><td>ec8 : [h2 → h1, w1 → w3, v2 → v3]</td><td>VL</td><td>0.09</td></tr></table>

**Insight.** For non-severe hardware changes, we can linearly transfer performance models across environments.

**Insight.** The strength of the influence of configuration options is typically preserved across environments.

**Insight.** A large percentage of configurations are typically invalid in both source and target environments.

<figure><img src="image_411.png" /></figure>
# Transfer Learning for Software Performance Analysis: An Exploratory Analysis
## Jamshidi et al. ASE 2017

SPEAR (SAT Solver)

Analysis time

14 options
16,384 configurations
SAT problems
3 hardware
2 versions

X264 (video encoder)

Encoding time

16 options
4,000 configurations
Video quality/size
2 hardware
3 versions

SQLite (DB engine)

Query time

14 options
1,000 configurations
DB Queries
2 hardware
2 versions

SaC (Compiler)

Execution time

50 options
71,267 configurations
10 Demo programs

<table><tr><td>ec1 : [h2 → h1, w3, v3]</td><td>SM</td><td>0.97</td></tr><tr><td>ec2 : [h2 → h1, w1, v3]</td><td>S</td><td>0.96</td></tr><tr><td>ec3 : [h1, w1 → w2, v3]</td><td>M</td><td>0.65</td></tr><tr><td>ec4 : [h1, w1 → w3, v3]</td><td>M</td><td>0.67</td></tr><tr><td>ec5 : [h1, w3, v2 → v3]</td><td>L</td><td>0.05</td></tr><tr><td>ec6 : [h1, w3, v1 → v3]</td><td>L</td><td>0.06</td></tr><tr><td>ec7 : [h1, w1 → w3, v2 → v3]</td><td>L</td><td>0.08</td></tr><tr><td>ec8 : [h2 → h1, w1 → w3, v2 → v3]</td><td>VL</td><td>0.09</td></tr></table>

**Insight.** For non-severe hardware changes, we can linearly transfer performance models across environments.

**Insight.** The strength of the influence of configuration options is typically preserved across environments.

**Insight.** A large percentage of configurations are typically invalid in both source and target environments.

<figure><img src="image_442.png" /></figure>
Transfer Learning for Software Performance Analysis: An Exploratory Analysis
Jamshidi et al. ASE 2017

SPEAR (SAT Solver)

Analysis time

14 options
16,384 configurations
SAT problems
3 hardware
2 versions

X264 (video encoder)

Encoding time

16 options
4,000 configurations
Video quality/size
2 hardware
3 versions

SQLite (DB engine)

Query time

14 options
1,000 configurations
DB Queries
2 hardware
2 versions

SaC (Compiler)

Execution time

50 options
71,267 configurations
10 Demo programs

<table><tr><td>ec1 : [h2 → h1, w3, v3]</td><td>SM</td><td>0.97</td></tr><tr><td>ec2 : [h2 → h1, w1, v3]</td><td>S</td><td>0.96</td></tr><tr><td>ec3 : [h1, w1 → w2, v3]</td><td>M</td><td>0.65</td></tr><tr><td>ec4 : [h1, w1 → w3, v3]</td><td>M</td><td>0.67</td></tr><tr><td>ec5 : [h1, w3, v2 → v3]</td><td>L</td><td>0.05</td></tr><tr><td>ec6 : [h1, w3, v1 → v3]</td><td>L</td><td>0.06</td></tr><tr><td>ec7 : [h1, w1 → w3, v2 → v3]</td><td>L</td><td>0.08</td></tr><tr><td>ec8 : [h2 → h1, w1 → w3, v2 → v3]</td><td>VL</td><td>0.09</td></tr></table>

<figure><img src="image_412.png" /></figure>

**mixing deep variability:** hard to assess the specific influence of each layer

**very few hardware, version, and input data... but lots of runtime configurations (variants)**

**Let's go deep with **input data!****
Transfer Learning for Software Performance Analysis: An Exploratory Analysis

Jamshidi et al. ASE 2017

SPEAR (SAT Solver)

Analysis time

14 options
16,384 configurations
SAT problems
3 hardware
2 versions

X264 (video encoder)

Encoding time

16 options
4,000 configurations
Video quality/size
2 hardware
3 versions

SQLite (DB engine)

Query time

14 options
1,000 configurations
DB Queries
2 hardware
2 versions

SaC (Compiler)

Execution time

50 options
71,267 configurations
10 Demo programs

<table><tr><td>ec1 : [h<sub>2</sub> → h<sub>1</sub>, w<sub>3</sub>, v<sub>3</sub>]</td><td>SM</td><td>0.97</td></tr><tr><td>ec2 : [h<sub>2</sub> → h<sub>1</sub>, w<sub>1</sub>, v<sub>3</sub>]</td><td>S</td><td>0.96</td></tr><tr><td>ec3 : [h<sub>1</sub>, w<sub>1</sub> → w<sub>2</sub>, v<sub>3</sub>]</td><td>M</td><td>0.65</td></tr><tr><td>ec4 : [h<sub>1</sub>, w<sub>1</sub> → w<sub>3</sub>, v<sub>3</sub>]</td><td>M</td><td>0.67</td></tr><tr><td>ec5 : [h<sub>1</sub>, w<sub>3</sub>, v<sub>2</sub> → v<sub>3</sub>]</td><td>L</td><td>0.05</td></tr><tr><td>ec6 : [h<sub>1</sub>, w<sub>3</sub>, v<sub>1</sub> → v<sub>3</sub>]</td><td>L</td><td>0.06</td></tr><tr><td>ec7 : [h<sub>1</sub>, w<sub>1</sub> → w<sub>3</sub>, v<sub>2</sub> → v<sub>3</sub>]</td><td>L</td><td>0.08</td></tr><tr><td>ec8 : [h<sub>2</sub> → h<sub>1</sub>, w<sub>1</sub> → w<sub>3</sub>, v<sub>2</sub> → v<sub>3</sub>]</td><td>VL</td><td>0.09</td></tr></table>

**mixing deep variability:** hard to assess the specific influence of each layer

**very few hardware, version, and input data... but lots of runtime configurations (variants)**

**Let's go deep with **input data!****

<figure><img src="image_443.png" /></figure>
# Practical impacts for users, developers, scientists, and self-adaptive systems

Threats to **variability knowledge** for performance property bitrate

<figure><img src="image_413.png" /></figure>

* optimal configuration is specific to an input; a good configuration can be a bad one

* some options' values have an opposite effect depending on the input

* effectiveness of sampling strategies (random, 2-wise, etc.) is input specific (somehow confirming Pereira et al. ICPE 2020)

* predicting, tuning, or understanding configurable systems without being aware of inputs can be inaccurate and... pointless

<figure><img src="image_414.png" /></figure>

<figure><img src="image_415.png" /></figure>
# Practical impacts for users, developers, scientists, and self-adaptive systems

Threats to **variability knowledge** for performance property bitrate

<figure><img src="image_444.png" /></figure>

* optimal configuration is specific to an input; a good configuration can be a bad one

* some options' values have an opposite effect depending on the input

* effectiveness of sampling strategies (random, 2-wise, etc.) is input specific (somehow confirming Pereira et al. ICPE 2020)

* predicting, tuning, or understanding configurable systems without being aware of inputs can be inaccurate and... pointless

<figure><img src="image_445.png" /></figure>

<figure><img src="image_446.png" /></figure>
# Computational science depends on software and its engineering

<figure><img src="image_416.png" /></figure>

- multi-million line of code base

- multi-dependencies

- multi-systems

- multi-layer

- multi-version

- multi-person

- multi-variant

## Climate is changing

<figure><img src="image_417.png" /></figure>

Observed warming 2009 - 2019 relative to 1961 - 1990

<figure><img src="image_418.png" /></figure>

<figure><img src="image_419.png" /></figure>

<figure><img src="image_420.png" /></figure>

## The sea level is rising

<figure><img src="image_421.png" /></figure>

<figure><img src="image_422.png" /></figure>

<figure><img src="image_423.png" /></figure>

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses
# Computational science depends on software and its engineering

<figure><img src="image_447.png" /></figure>

- multi-million line of code base

- multi-dependencies

- multi-systems

- multi-layer

- multi-version

- multi-person

- multi-variant

## Climate is changing

<figure><img src="image_448.png" /></figure>

Observed warming 2009 - 2019 relative to 1961 - 1990

<figure><img src="image_449.png" /></figure>

<figure><img src="image_450.png" /></figure>

<figure><img src="image_451.png" /></figure>

## The sea level is rising

<figure><img src="image_452.png" /></figure>

<figure><img src="image_453.png" /></figure>

<figure><img src="image_454.png" /></figure>

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses
# x264 video encoder (compilation/build)

```
x264 --bframes 1 --ref 3 --cabac DiverSE-meeting.mp4 -o meeting13.webm
```

mathieuacher

localhost.localdomain

.../x264-SLIMFAST/x264

x264-gcov

./configure --help

<table><tr><td>--disable-thread</td><td>disable multithreaded encoding</td></tr><tr><td>--disable-win32thread</td><td>disable win32threads (windows only)</td></tr><tr><td>--disable-interlaced</td><td>disable interlaced encoding support</td></tr><tr><td>--bit-depth=BIT_DEPTH</td><td>set output bit depth (8, 10, all) [all]</td></tr><tr><td>--chroma-format=FORMAT</td><td>output chroma format (400, 420, 422, 444, all) [all]</td></tr></table>

Advanced options:

<table><tr><td>--disable-asm</td><td>disable platform-specific assembly optimizations</td></tr><tr><td>--enable-lto</td><td>enable link-time optimization</td></tr><tr><td>--enable-debug</td><td>add -g</td></tr><tr><td>--enable-gprof</td><td>add -pg</td></tr><tr><td>--enable-strip</td><td>add -s</td></tr><tr><td>--enable-pic</td><td>build position-independent code</td></tr></table>

Cross-compilation:

<table><tr><td>--host=HOST</td><td>build programs to run on HOST</td></tr><tr><td>--cross-prefix=PREFIX</td><td>use PREFIX for compilation tools</td></tr><tr><td>--sysroot=SYSROOT</td><td>root of cross-build tree</td></tr></table>

External library support:

<table><tr><td>--disable-avs</td><td>disable avisynth support</td></tr><tr><td>--disable-swscale</td><td>disable swscale support</td></tr><tr><td>--disable-lavf</td><td>disable lavformat support</td></tr><tr><td>--disable-ffms</td><td>disable fmmpegsource support</td></tr><tr><td>--disable-gpac</td><td>disable gpac support</td></tr><tr><td>--disable-lsmash</td><td>disable lsmash support</td></tr></table>

compile-time
options

<figure><img src="image_424.png" /></figure>
# x264 video encoder (compilation/build)

~ x264 --bframes 1 --ref 3 --cabac DiverSE-meeting.mp4 -o meeting13.webm

mathieuacher

localhost.localdomain

../x264-SLIMFAST/x264

x264-gcov

<figure><img src="image_455.png" /></figure>

./configure --help

<table><tr><td>--disable-thread</td><td>disable multithreaded encoding</td></tr><tr><td>--disable-win32thread</td><td>disable win32threads (windows only)</td></tr><tr><td>--disable-interlaced</td><td>disable interlaced encoding support</td></tr><tr><td>--bit-depth=BIT_DEPTH</td><td>set output bit depth (8, 10, all) [all]</td></tr><tr><td>--chroma-format=FORMAT</td><td>output chroma format (400, 420, 422, 444, all) [all]</td></tr></table>

Advanced options:

<table><tr><td>--disable-asm</td><td>disable platform-specific assembly optimizations</td></tr><tr><td>--enable-lto</td><td>enable link-time optimization</td></tr><tr><td>--enable-debug</td><td>add -g</td></tr><tr><td>--enable-gprof</td><td>add -pg</td></tr><tr><td>--enable-strip</td><td>add -s</td></tr><tr><td>--enable-pic</td><td>build position-independent code</td></tr></table>

Cross-compilation:

<table><tr><td>--host=HOST</td><td>build programs to run on HOST</td></tr><tr><td>--cross-prefix=PREFIX</td><td>use PREFIX for compilation tools</td></tr><tr><td>--sysroot=SYSROOT</td><td>root of cross-build tree</td></tr></table>

External library support:

<table><tr><td>--disable-avs</td><td>disable avisynth support</td></tr><tr><td>--disable-swscale</td><td>disable swscale support</td></tr><tr><td>--disable-lavf</td><td>disable libavformat support</td></tr><tr><td>--disable-ffms</td><td>disable fmmpegsource support</td></tr><tr><td>--disable-gpac</td><td>disable gpac support</td></tr><tr><td>--disable-lsmash</td><td>disable lsmash support</td></tr></table>

compile-time
options

<figure><img src="image_456.png" /></figure>
# What can we do? (#1 studies)

Empirical studies about deep software variability

* more subject systems

* more variability layers, including interactions

* more quantitative (e.g., performance) properties

with challenges for gathering measurements data:

* how to scale experiments? Variant space is huge!

* how to fix/isolate some layers? (eg hardware)

* how to measure in a reliable way?

Expected outcomes:

* significance of deep software variability in the wild

* identification of **stable** layers: sources of variability that should not affect the conclusion and that can be eliminated/forgotten

* identification/quantification of **sensitive** layers and interactions that matter

* **variability knowledge**

<figure><img src="image_425.png" /></figure>
# What can we do? (#1 studies)

Empirical studies about deep software variability

* more subject systems

* more variability layers, including interactions

* more quantitative (e.g., performance) properties

with challenges for gathering measurements data:

* how to scale experiments? Variant space is huge!

* how to fix/isolate some layers? (eg hardware)

* how to measure in a reliable way?

Expected outcomes:

* significance of deep software variability in the wild

* identification of **stable** layers: sources of variability that should not affect the conclusion and that can be eliminated/forgotten

* identification/quantification of **sensitive** layers and interactions that matter

* **variability knowledge**

<figure><img src="image_457.png" /></figure>
# What can we do? (#2 cost)

## Reducing the cost of exploring the variability spaces

Many directions here (references at the end of the slides):

* **learning**
  * many algorithms/techniques with tradeoffs interpretability/accuracy
  * transfer learning (instead of learning from scratch)

* **sampling strategies**
  * uniform random sampling? t-wise? distance-based? ...
  * sample of hardware? input data?

* incremental build of configurations

* white-box approaches

* ...

<figure><img src="image_426.png" /></figure>
# What can we do? (#2 cost)

## Reducing the cost of exploring the variability spaces

Many directions here (references at the end of the slides):

* **learning**
  * many algorithms/techniques with tradeoffs interpretability/accuracy
  * transfer learning (instead of learning from scratch)

* **sampling strategies**
  * uniform random sampling? t-wise? distance-based? ...
  * sample of hardware? input data?

* incremental build of configurations

* white-box approaches

* ...

<figure><img src="image_458.png" /></figure>
# Key results (for x264)

<figure><img src="image_427.png" /></figure>

**Worth tuning software at compile-time:** gain about 10 % of execution time with the tuning of compile-time options (compared to the default compile-time configuration). The improvements can be larger for some inputs and some runtime configurations.

**Stability of variability knowledge:** For all the execution time distributions of x264 and all the input videos, the worst correlation is greater than 0.97. If the compile-time options change the scale of the distribution, they do not change the rankings of run-time configurations (i.e., they do not truly interact with the run-time options).

**Reuse of configuration knowledge:** $f_1 = \beta \times f_2 + \alpha$

* Linear transformation among distributions

* Users can also trust the documentation of run-time options, consistent whatever the compile-time configuration is.

<figure><img src="image_428.png" /></figure>
# Key results (for x264)

<figure><img src="image_459.png" /></figure>

**Worth tuning software at compile-time:** gain about 10 % of execution time with the tuning of compile-time options (compared to the default compile-time configuration). The improvements can be larger for some inputs and some runtime configurations.

**Stability of variability knowledge:** For all the execution time distributions of x264 and all the input videos, the worst correlation is greater than 0.97. If the compile-time options change the scale of the distribution, they do not change the rankings of run-time configurations (i.e., they do not truly interact with the run-time options).

**Reuse of configuration knowledge:** $f_1 = \beta \times f_2 + \alpha$

* Linear transformation among distributions

* Users can also trust the documentation of run-time options, consistent whatever the compile-time configuration is.

<figure><img src="image_460.png" /></figure>
# Our Vision

Embrace deep variability!

Explicit modeling of the variability points and their relationships, such as:

1. Get insights into the variability “factors” and their possible interactions

2. Capture and document configurations for the sake of **reproducibility**

3. Explore diverse configurations to **replicate**, and hence optimize, validate, increase the robustness, or provide better resilience

https://hal.science/hal-04582287

## Embracing Deep Variability For Reproducibility and Replicability

Mathieu Acher

Univ Rennes, Inria, CNRS, IRISA, IUF
Rennes, France

Benoit Combemale

Univ Rennes, Inria, CNRS, IRISA
Rennes, France

Jean-Marc Jézéquel

Univ Rennes, Inria, CNRS, IRISA, IUF
Rennes, France

Georges Aaron Randrianaina

Univ Rennes, Inria, CNRS, IRISA
Rennes, France

### ABSTRACT

Reproducibility (a.k.a. determinism in some cases) constitutes a fundamental aspect in various fields of computer science, such as floating-point computations in numerical analysis and simulation, concurrency models in parallelism, reproducible builds for third parties integration and packaging, and containerization for execution environments. These concepts, while pervasive across diverse concerns, often exhibit intricate inter-dependencies, making it challenging to achieve a comprehensive understanding. In this short and vision paper we delve into the application of software engineering techniques, specifically variability management, to systematically identify and explicit points of variability that may give rise to reproducibility issues (e.g. language, libraries, compiler, virtual machine, OS, environment variables, etc). The primary objectives are: i) gaining insights into the variability layers and their possible interactions, ii) capturing and documenting configurations for the sake of reproducibility, and iii) exploring diverse configurations to replicate, and hence validate and ensure the robustness of results. By adopting these methodologies, we aim to address the complexities associated with reproducibility and replicability in modern software systems and environments, facilitating a more comprehensive and nuanced perspective on these critical aspects.

### 1 INTRODUCTION

Many scientific domains need to process large amount of data with more and more complex computations. For instance, studies about climate modelling and change involve the design of mathematical model, the mining and analysis of data, the executions of large simulations, etc. [16, 24, 29]. These computational tasks rely on various kinds of software, from a set of scripts to automate the deployment to a comprehensive system containing several features that help researchers exploring various hypotheses. It is not an overstatement to say that computational science depends on software and its engineering [2, 34, 54].

One of the main promise of software is that a result obtained by an experiment (e.g. simulation) can be achieved again with a high degree of agreement. But despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results [6, 9, 15, 19, 22, 31, 41, 42, 53]. For instance, applications of different analysis pipelines, alterations

community [22]. As a result, software can threaten the scientific knowledge and recommendations built on top of these computations and studies.

In this paper we propose to characterize both intended and unintended variability of any software-intensive system in order to support reproducibility and replicability, and eventually estimate its robustness, uncertainty profile, and explore different hypotheses.

### 2 DEEP SOFTWARE VARIABILITY

Uncertainty in informatics comes from many different origins [17, 39], either ontological (i.e., inherent unpredictability, e.g., aleatory) or epistemic (i.e., due to insufficient knowledge).

Ontological causes include noise in the input data of a program, its memory layout, network delays, the internal state of the processor, the ambient temperature and even the age of the processor¹.

Epistemic causes include misunderstanding of the user's needs, variable behavior of conceptually similar resolution methods, choice of threshold parameters, unexpected behavior of APIs, variable behavior among functionally similar libraries, or subtle differences in the semantics of programming languages (e.g., ~3% evaluates to ~1 in Java but 1 in Python), or even inside the same programming language (for instance x/0 is an undefined behavior in C).

<figure><img src="image_429.png" /><figcaption>Figure 1: Deep Variability</figcaption></figure>

⇒ We aim to address the complexities associated with reproducibility and replicability in modern software systems and environments, facilitating a more comprehensive and nuanced perspective on these critical “factors”.
# Our Vision

Embrace deep variability!

Explicit modeling of the variability points and their relationships, such as:

1. Get insights into the variability “factors” and their possible interactions

2. Capture and document configurations for the sake of **reproducibility**

3. Explore diverse configurations to **replicate**, and hence optimize, validate, increase the robustness, or provide better resilience

https://hal.science/hal-04582287

## Embracing Deep Variability For Reproducibility & Replicability

Mathieu Acher

Univ Rennes, Inria, CNRS, IRISA, IUF
Rennes, France

Benoit Combemale

Univ Rennes, Inria, CNRS, IRISA
Rennes, France

Jean-Marc Jézéquel

Univ Rennes, Inria, CNRS, IRISA, IUF
Rennes, France

Georges Aaron Randrianaina

Univ Rennes, Inria, CNRS, IRISA
Rennes, France

### ABSTRACT

Reproducibility (a.k.a. determinism in some cases) constitutes a fundamental aspect in various fields of computer science, such as floating-point computations in numerical analysis and simulation, concurrency models in parallelism, reproducible builds for third parties integration and packaging, and containerization for execution environments. These concepts, while pervasive across diverse concerns, often exhibit intricate inter-dependencies, making it challenging to achieve a comprehensive understanding. In this short and vision paper we delve into the application of software engineering techniques, specifically variability management, to systematically identify and explicit points of variability that may give rise to reproducibility issues (e.g. language, libraries, compiler, virtual machine, OS, environment variables, etc). The primary objectives are: i) gaining insights into the variability layers and their possible interactions, ii) capturing and documenting configurations for the sake of reproducibility, and iii) exploring diverse configurations to replicate, and hence validate and ensure the robustness of results. By adopting these methodologies, we aim to address the complexities associated with reproducibility and replicability in modern software systems and environments, facilitating a more comprehensive and nuanced perspective on these critical aspects.

### 1 INTRODUCTION

Many scientific domains need to process large amount of data with more and more complex computations. For instance, studies about climate modelling and change involve the design of mathematical model, the mining and analysis of data, the executions of large simulations, etc. [16, 24, 29]. These computational tasks rely on various kinds of software, from a set of scripts to automate the deployment to a comprehensive system containing several features that help researchers exploring various hypotheses. It is not an overstatement to say that computational science depends on software and its engineering [2, 34, 54].

One of the main promise of software is that a result obtained by an experiment (e.g. simulation) can be achieved again with a high degree of agreement. But despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results [6, 9, 15, 19, 22, 31, 41, 42, 53]. For instance, applications of different analysis pipelines, alterations

community [22]. As a result, software can threaten the scientific knowledge and recommendations built on top of these computations and studies.

In this paper we propose to characterize both intended and unintended variability of any software-intensive system in order to support reproducibility and replicability, and eventually estimate its robustness, uncertainty profile, and explore different hypotheses.

### 2 DEEP SOFTWARE VARIABILITY

Uncertainty in informatics comes from many different origins [17, 39], either ontological (i.e., inherent unpredictability, e.g., aleatory) or epistemic (i.e., due to insufficient knowledge).

Ontological causes include noise in the input data of a program, its memory layout, network delays, the internal state of the processor, the ambient temperature and even the age of the processor¹.

Epistemic causes include misunderstanding of the user's needs, variable behavior of conceptually similar resolution methods, choice of threshold parameters, unexpected behavior of APIs, variable behavior among functionally similar libraries, or subtle differences in the semantics of programming languages (e.g., ~3% evaluates to ~1 in Java but 1 in Python), or even inside the same programming language (for instance x/0 is an undefined behavior in C).

<figure><img src="image_461.png" /><figcaption>Figure 1: Deep Variability</figcaption></figure>

⇒ We aim to **address the complexities associated with reproducibility and replicability in modern software systems and environments**, facilitating a more comprehensive and nuanced perspective on these critical “factors”.
Multi-Level Analysis of Compiler-Induced Variability and
Performance Tradeoffs

<table><tr><td>Michael Bentley</td><td>Dong H. Ahn</td></tr><tr><td> Ian Briggs</td><td> Ignacio Laguna</td></tr><tr><td> Ganesh Gopalakrishnan</td><td> Holger E. Jones</td></tr><tr><td> mbentley@cs.utah.edu</td><td> ahn1@llnl.gov</td></tr><tr><td> ianbriggsutah@gmail.com</td><td> lagunaperalt1@llnl.gov</td></tr><tr><td> ganesh@cs.utah.edu</td><td> lee218@llnl.gov</td></tr><tr><td> University of Utah</td><td> jones19@llnl.gov</td></tr><tr><td></td><td> Lawrence Livermore National Laboratory</td></tr></table>

**Definition of Reproducibility.** Given the growing heterogeneity of hardware and software, one cannot always define reproducibility as achieving bitwise reproducible results. Instead, we view a reproducible computation as one that produces a result within an “acceptable range” of a trusted baseline answer. In FLiT, we rely on the application developer to provide an acceptance testing function that (indirectly) defines this range.

<table><caption>Table 1: Compilers used in the MFEM study with summary statistics. The best flags are chosen by the best average speedup across all MFEM examples. The average speedup over all 19 MFEM examples is reported and is calculated relative to the speed of g++ -O2.</caption><thead><tr><td>Compiler</td><td>Released</td><td># Variable Runs</td><td>Best Flags</td><td>Speedup</td></tr></thead><tbody><tr><td>gcc-8.2.0</td><td>26 July 2018</td><td>78 of 1,288 (6.0%)</td><td>-02 -funsafe-math-optimizations</td><td>1.097</td></tr><tr><td>clang-6.0.1</td><td>05 July 2018</td><td>24 of 1,368 (1.8%)</td><td>-03 -funsafe-math-optimizations</td><td>1.042</td></tr><tr><td>icpc-18.0.3</td><td>16 May 2018</td><td>984 of 1,976 (49.8%)</td><td>-02 -fp-model fast=2</td><td>1.056</td></tr></tbody></table>

<table><tbody><tr><td>-fpc-fortran-rules</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fpc-limit-range</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fexcess-precision-fast</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fexcess-precision-standard</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-ffinite-math-only</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-ffloat-store</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fp-contract-on</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fna</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-merge-all-constants</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fno-trapping-math</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fp-model fast=1</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model fast=2</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model double</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model=extended</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model=precise</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model=source</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model=strict</td><td></td><td></td><td>X</td></tr><tr><td>-fp-port</td><td></td><td></td><td>X</td></tr><tr><td>-freeprocal-math</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-frounding-math</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fakingmath-nans</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fsingle-precision-constant</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fusafe-math-optimizations</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-march=core-avx</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-navx</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-navx2-fna</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fpmath=ase -stune=native</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-spt</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-no-fna</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-no-fit</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-no-prec-div</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-prew-div</td><td></td><td></td><td>X</td></tr><tr><td>-fnaad-false</td><td></td><td></td><td>X</td></tr><tr><td>-fnaad=true</td><td></td><td></td><td>X</td></tr><tr><td>-fis2-true</td><td></td><td></td><td>X</td></tr><tr><td>-prew-div-false</td><td></td><td></td><td>X</td></tr><tr><td>-prew-div-true</td><td></td><td></td><td>X</td></tr><tr><td>-prew-sqrt-false</td><td></td><td></td><td>X</td></tr><tr><td>-prew-sqrt-true</td><td></td><td></td><td>X</td></tr><tr><td>-use_fast</td><td>X</td><td>X</td><td>X</td></tr></tbody></table>

<figure><img src="image_430.png" /><figcaption>Figure 1: Multi-level workflow. Levels are (1) determine variability-inducing compilations, (2) analyze the space of reproducibility and performance, and (3) debug variability by identifying files and functions causing variability.</figcaption></figure>
Multi-Level Analysis of Compiler-Induced Variability and
Performance Tradeoffs

<table><tr><td>Michael Bentley</td><td>Dong H. Ahn</td></tr><tr><td>Ian Briggs</td><td>Ignacio Laguna</td></tr><tr><td>Ganesh Gopalakrishnan</td><td>Gregory L. Lee</td></tr><tr><td>mbentley@cs.utah.edu</td><td>Holger E. Jones</td></tr><tr><td>iandbriggsutah@gmail.com</td><td>ahn1@llnl.gov</td></tr><tr><td>ganesh@cs.utah.edu</td><td>lagunaperalt1@llnl.gov</td></tr><tr><td>University of Utah</td><td>lee218@llnl.gov</td></tr><tr><td></td><td>jones19@llnl.gov</td></tr><tr><td></td><td>Lawrence Livermore National Laboratory</td></tr></table>

**Definition of Reproducibility.** Given the growing heterogeneity of hardware and software, one cannot always define reproducibility as achieving bitwise reproducible results. Instead, we view a reproducible computation as one that produces a result within an “acceptable range” of a trusted baseline answer. In FLiT, we rely on the application developer to provide an acceptance testing function that (indirectly) defines this range.

<table><caption>Table 1: Compilers used in the MFEM study with summary statistics. The best flags are chosen by the best average speedup across all MFEM examples. The average speedup over all 19 MFEM examples is reported and is calculated relative to the speed of g++ -O2.</caption><thead><tr><td>Compiler</td><td>Released</td><td># Variable Runs</td><td>Best Flags</td><td>Speedup</td></tr></thead><tbody><tr><td>gcc-8.2.0</td><td>26 July 2018</td><td>78 of 1,288 (6.0%)</td><td>-02 -funsafe-math-optimizations</td><td>1.097</td></tr><tr><td>clang-6.0.1</td><td>05 July 2018</td><td>24 of 1,368 (1.8%)</td><td>-03 -funsafe-math-optimizations</td><td>1.042</td></tr><tr><td>icpc-18.0.3</td><td>16 May 2018</td><td>984 of 1,976 (49.8%)</td><td>-02 -fp-model fast=2</td><td>1.056</td></tr></tbody></table>

<table><tr><td>-fpc-fortran-rules</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fpc-limit-range</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fexcess-precision-fast</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fexcess-precision-standard</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-ffinite-math-only</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-ffloat-store</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fp-contract-on</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fna</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-merge-all-constants</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fno-trapping-math</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fp-model fast=1</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model fast=2</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model double</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model=extended</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model=precise</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model=source</td><td></td><td></td><td>X</td></tr><tr><td>-fp-model=strict</td><td></td><td></td><td>X</td></tr><tr><td>-fp-port</td><td></td><td></td><td>X</td></tr><tr><td>-freeprocal-math</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-frounding-math</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fraising-math</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fsingle-precision-constant</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fstrict</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-funsafe-math-optimizations</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-march=core-avx</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-mavx</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-mavx2 -mfn</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-fpmath=ase -stune=native</td><td>X</td><td>X</td><td>X</td></tr><tr><td>-spt</td><td></td><td></td><td>X</td></tr><tr><td>-no-fma</td><td></td><td></td><td>X</td></tr><tr><td>-no-fix</td><td></td><td></td><td>X</td></tr><tr><td>-no-prec-div</td><td></td><td></td><td>X</td></tr><tr><td>-prew-div</td><td></td><td></td><td>X</td></tr><tr><td>-fnaad-false</td><td></td><td></td><td>X</td></tr><tr><td>-fnaad=true</td><td></td><td></td><td>X</td></tr><tr><td>-fis2-true</td><td></td><td></td><td>X</td></tr><tr><td>-prew-div-false</td><td></td><td></td><td>X</td></tr><tr><td>-prew-div-true</td><td></td><td></td><td>X</td></tr><tr><td>-prew-sqrt-false</td><td></td><td></td><td>X</td></tr><tr><td>-prew-sqrt-true</td><td></td><td></td><td>X</td></tr><tr><td>-use_fast</td><td></td><td></td><td>X</td></tr></table>

<figure><img src="image_462.png" /><figcaption>Figure 1: Multi-level workflow. Levels are (1) determine variability-inducing compilations, (2) analyze the space of reproducibility and performance, and (3) debug variability by identifying files and functions causing variability.</figcaption></figure>
`exec(software) = exec_repro(software)`

or

`exec(software) =~ exec(software_repro)`

(difference: `exec_repro` is another execution environment... and so somehow differs or not with `exec`; or we consider that software differs...)

(`exec`: execution? what's the outcome then? in fact execution can be replaced by "build"... which is another kind of execution)

`exec(software) ?= exec_repro(software)`

`software =~ software_repro`

`exec(software, hardware)`

`exec(software, hardware, compiler, input_data, operating_system, bios, container, hypervisor, dependencies_versions)`

`exec (v1, v2, ..., vN) =~ exec_repro (v1', v2', ..., vN')`

`for i in [1, n], v_{i} =~ v_{i} (or not!)`

`~= is specific to a domain, to a usage, etc.`

`~= can be over the N layers or over N' layers (N' < N)`

`~= can be specific to some pairs elements (eg we know that with this hardware, the exec time is multiplied by 2)`

for instance, we know the `~=` between a software configuration with any hardware (but if the compiler changes, then the `~=` should be "tuned" accordingly)

also `~=` can be defined between a configuration set and an hardware set (eg performance distribution)
`exec(software) = exec_repro(software)`

or

`exec(software) =~ exec(software_repro)`

*(difference: `exec_repro` is another execution environment... and so somehow differs or not with `exec`; or we consider that software differs...)*

*(`exec`: execution? what's the outcome then? in fact execution can be replaced by "build"... which is another kind of execution)*

`exec(software) ?= exec_repro(software)`

`software =~ software_repro`

`exec(software, hardware)`

`exec(software, hardware, compiler, input_data, operating_system, bios, container, hypervisor, dependencies_versions)`

`exec (v1, v2, ..., vN) =~ exec_repro (v1', v2', ..., vN')`

`for i in [1, n], v_{i} =~ v_{i} (or not!)`

`~= is specific to a domain, to a usage, etc.`

`~= can be over the N layers or over N' layers (N' < N)`

`~= can be specific to some pairs elements (eg we know that with this hardware, the exec time is multiplied by 2)`

for instance, we know the `~=` between a software configuration with any hardware (but if the compiler changes, then the `~=` should be "tuned" accordingly)

also `~=` can be defined between a configuration set and an hardware set (eg performance distribution)
## Exact same results? No

**Definition of Reproducibility.** Given the growing heterogeneity of hardware and software, one cannot always define reproducibility as achieving bitwise reproducible results. Instead, we view a reproducible computation as one that produces a result within an “acceptable range” of a trusted baseline answer. In FLiT, we rely on the application developer to provide an acceptance testing function that (indirectly) defines this range.

## Multi-Level Analysis of Compiler-Induced Variability and Performance Tradeoffs

Michael Bentley
Ian Briggs
Ganesh Gopalakrishnan
mbentley@cs.utah.edu
ianbriggsutah@gmail.com
ganesh@cs.utah.edu
University of Utah

Dong H. Ahn
Ignacio Laguna
Gregory L. Lee
Holger E. Jones
ahn1@llnl.gov
lagunaperalt1@llnl.gov
lee218@llnl.gov
jones19@llnl.gov
Lawrence Livermore National Laboratory

<figure><img src="image_431.png" /><figcaption>Table 1: Compilers used in the MFEM study with summary statistics. The best flags are chosen by the best average speedup across all MFEM examples. The average speedup over all 19 MFEM examples is reported and is calculated relative to the speedup of g** -02.</figcaption></figure>

<table><caption>Figure 1: Multi-level workflow. Levels are (1) determine variability-inducing compilations, (2) analyze the space of reproducible and performance, and (3) debug variability by identifying files and functions causing variability.</caption><thead><tr><td>Compiler</td><td>Released</td><td># Variable Runs</td><td>Best Flags</td><td>Speedup</td></tr></thead><tbody><tr><td>gcc-8.2.0</td><td>26 July 2018</td><td>78 of 1,288 (6.0%)</td><td>-02 -funsafe-math-optimizations</td><td>1.097</td></tr><tr><td>clang-6.0.1</td><td>05 July 2018</td><td>24 of 1,368 (1.8%)</td><td>-03 -funsafe-math-optimizations</td><td>1.042</td></tr><tr><td>icp-18.0.3</td><td>16 May 2018</td><td>984 of 1,978 (49.8%)</td><td>-02 -fp-model fast-2</td><td>1.056</td></tr></tbody></table>
## Exact same results? No

**Definition of Reproducibility.** Given the growing heterogeneity of hardware and software, one cannot always define reproducibility as achieving bitwise reproducible results. Instead, we view a reproducible computation as one that produces a result within an “acceptable range” of a trusted baseline answer. In FLiT, we rely on the application developer to provide an acceptance testing function that (indirectly) defines this range.

## Multi-Level Analysis of Compiler-Induced Variability and Performance Tradeoffs

Michael Bentley
Ian Briggs
Ganesh Gopalakrishnan
mbentley@cs.utah.edu
ianbriggsutah@gmail.com
ganesh@cs.utah.edu
University of Utah

Dong H. Ahn
Ignacio Laguna
Gregory L. Lee
Holger E. Jones
ahn1@llnl.gov
lagunaperalt1@llnl.gov
lee218@llnl.gov
jones19@llnl.gov
Lawrence Livermore National Laboratory

<figure><img src="image_463.png" /><figcaption>Table 1: Compilers used in the MFEM study with summary statistics. The best flags are chosen by the best average speedup across all MFEM examples. The average speedup over all 19 MFEM examples is reported and is calculated relative to the speedup of g** -02.</figcaption></figure>

<table><caption>Figure 1: Multi-level workflow. Levels are (1) determine variability-inducing compilations, (2) analyze the space of reproducible and performance, and (3) debug variability by identifying files and functions causing variability.</caption><thead><tr><td>Compiler</td><td>Released</td><td># Variable Runs</td><td>Best Flags</td><td>Speedup</td></tr></thead><tbody><tr><td>gcc-8.2.0</td><td>26 July 2018</td><td>78 of 1,288 (6.0%)</td><td>-02 -funsafe-math-optimizations</td><td>1.097</td></tr><tr><td>clang-6.0.1</td><td>05 July 2018</td><td>24 of 1,368 (1.8%)</td><td>-03 -funsafe-math-optimizations</td><td>1.042</td></tr><tr><td>icp-18.0.3</td><td>16 May 2018</td><td>984 of 1,978 (49.8%)</td><td>-02 -fp-model fast-2</td><td>1.056</td></tr></tbody></table>
Frictionless reproducibility (annotated bibliography; grey literature)

https://hdsr.mitpress.mit.edu/pub/8dqqwqiu/release/1 The Mechanics of Frictionless
Reproducibility, B Recht

interesting historical perspective on research in neural networks (NeurIPs 87 titles are shockingly
still relevant); really love some parts about random experiments, science as a “massively parallel
genetic algorithm” or the discussions on the difficulty of using ML/DL software (completely
aligned with my terrible experience of Weka GUI in ~2006)

https://www.argmin.net/p/the-department-of-frictionless-reproducibility

https://statmodeling.stat.columbia.edu/2023/10/13/frictionless-reproducibility-methods-as-proto-al
gorithms-division-of-labor-as-a-characteristic-of-statistical-methods-statistics-as-the-science-of-d
efaults-statisticians-well-prepared-to-think-abo/
# Frictionless reproducibility (annotated bibliography; grey literature)

https://hdsr.mitpress.mit.edu/pub/8dqqwqiu/release/1 The Mechanics of Frictionless
Reproducibility, B Recht

interesting historical perspective on research in neural networks (NeurIPs 87 titles are shockingly
still relevant); really love some parts about random experiments, science as a “massively parallel
genetic algorithm” or the discussions on the difficulty of using ML/DL software (completely
aligned with my terrible experience of Weka GUI in ~2006)

https://www.argmin.net/p/the-department-of-frictionless-reproducibility

https://statmodeling.stat.columbia.edu/2023/10/13/frictionless-reproducibility-methods-as-proto-al
gorithms-division-of-labor-as-a-characteristic-of-statistical-methods-statistics-as-the-science-of-d
efaults-statisticians-well-prepared-to-think-abo/
Progress and frictionless reproducibility

Inspired by Thomas Kuhn (1962), we can think of the scientific and engineering process as a massively parallel genetic algorithm. If we want to improve upon the systems we currently have, we might try a small perturbation to see if we get an improvement. If we can find a small change that improves some desired outcome, we could change our systems to reflect this improvement. If we continually search for these improvements and work hard to demonstrate their value, we may head in a better direction over time.

For scientific endeavors, we could perhaps gauge ‘better’ or ‘worse’ by performing random experiments—not randomized experiments per se, but random experiments in the sense of trying potentially surprising improvements. If our small tweak results in better outcomes, we can attempt to convince a journal editor or conference program committee to publish it. And this communication gives everyone else a new starting point for their own random experimentation.

A single investigator can only make so much progress by random searching alone, but random search is pleasantly parallelizable. Competing scientists can independently try their own random ideas and publish their results. Sometimes an individual result is so promising that the herd of experimenters all flock around the good idea, hoping to strike gold on a nearby improvement and bring home bragging rights. To some, this looks like an inefficient mess. To others, it looks like science.

https://hdsr.mitpress.mit.edu/pub/8dqqwqiu/release/1 The Mechanics of Frictionless
Reproducibility, B Recht
# Progress and frictionless reproducibility

Inspired by Thomas Kuhn (1962), we can think of the scientific and engineering process as a massively parallel genetic algorithm. If we want to improve upon the systems we currently have, we might try a small perturbation to see if we get an improvement. If we can find a small change that improves some desired outcome, we could change our systems to reflect this improvement. If we continually search for these improvements and work hard to demonstrate their value, we may head in a better direction over time.

For scientific endeavors, we could perhaps gauge 'better' or 'worse' by performing random experiments—not randomized experiments per se, but random experiments in the sense of trying potentially surprising improvements. If our small tweak results in better outcomes, we can attempt to convince a journal editor or conference program committee to publish it. And this communication gives everyone else a new starting point for their own random experimentation.

A single investigator can only make so much progress by random searching alone, but random search is pleasantly parallelizable. Competing scientists can independently try their own random ideas and publish their results. Sometimes an individual result is so promising that the herd of experimenters all flock around the good idea, hoping to strike gold on a nearby improvement and bring home bragging rights. To some, this looks like an inefficient mess. To others, it looks like science.

https://hdsr.mitpress.mit.edu/pub/8dqqwqiu/release/1 The Mechanics of Frictionless Reproducibility, B Recht
# Data sharing and frictions

> "Data set benchmarking and competitive testing took over machine learning in the late 1980s. Email and file transfer were becoming more accessible. The current specification of FTP was finalized in 1985. In 1987, a PhD student at UC Irvine named David Aha put up an FTP server to host data sets for empirically testing machine learning methods. Aha was motivated by service to the community, but he also wanted to show his nearest-neighbor methods would outperform Ross Quinlan's decision tree induction algorithms. He formatted his data sets using the 'attribute-value' representation that a rival researcher, Ross Quinlan (1986), had used. And, so, the UC Irvine Machine Learning Repository was born."

> "Improvements in computing greased the wheels, giving us faster computers, faster data transfer, and smaller storage footprints. But computing technology alone was not sufficient to drive progress. Friendly competition with Quinlan inspired Aha to build the UCI repository. **And more explicit competitions were also crucial components of the success.**"

The Mechanics of Frictionless Reproducibility, B Recht, 2024

https://hdsr.mitpress.mit.edu/pub/8dqgwqiu/release/1
# Data sharing and frictions

> "Data set benchmarking and competitive testing took over machine learning in the late 1980s. Email and file transfer were becoming more accessible. The current specification of FTP was finalized in 1985. In 1987, a PhD student at UC Irvine named David Aha put up an FTP server to host data sets for empirically testing machine learning methods. Aha was motivated by service to the community, but he also wanted to show his nearest-neighbor methods would outperform Ross Quinlan's decision tree induction algorithms. He formatted his data sets using the 'attribute-value' representation that a rival researcher, Ross Quinlan (1986), had used. And, so, the UC Irvine Machine Learning Repository was born."

> "Improvements in computing greased the wheels, giving us faster computers, faster data transfer, and smaller storage footprints. But computing technology alone was not sufficient to drive progress. Friendly competition with Quinlan inspired Aha to build the UCI repository. **And more explicit competitions were also crucial components of the success.**"

The Mechanics of Frictionless Reproducibility, B Recht, 2024

https://hdsr.mitpress.mit.edu/pub/8dqgwqiu/release/1
and up with a big divergence.

In [1]: import torch, struct
...: def binary_double(num):
...:     print("'.join(f'{{c:0>8b}}' for c in struct.pack('!d', num)))
...:     binary_double(torch.tensor(9, device='cuda')/10)
...:     binary_double(torch.tensor(9, device='cpu')/10)
001111111101100110011001100110011110000000000000000000000000000
001111111111011001100110011001100111100000000000000000000000000

<figure><img src="image_432.png" /></figure>

Stas Bekman

@StasBekman · Jan 20

Floating point math discrepancies with some pretrained LM models can be an issue.

I was debugging today a weird discrepancy between Llama-2-7b inference
results which proved to be triggered by whether `from_pretrained` was called ...

Show more

<figure><img src="image_433.png" /></figure>

This is from the mps device:

<figure><img src="image_434.png" /></figure>
and up with a big divergence.

In [1]: import torch, struct
...: def binary_double(num):
...:     print("'.join(f'{{c:0>8b}}' for c in struct.pack('!d', num)))
...:     binary_double(torch.tensor(9, device='cuda')/10)
...:     binary_double(torch.tensor(9, device='cpu')/10)
001111111101100110011001100110011110000000000000000000000000000
001111111111011001100110011001100111100000000000000000000000000

<figure><img src="image_464.png" /></figure>

Stas Bekman

@StasBekman · Jan 20

Floating point math discrepancies with some pretrained LM models can be an issue.

I was debugging today a weird discrepancy between Llama-2-7b inference
results which proved to be triggered by whether `from_pretrained` was called ...

Show more

<figure><img src="image_465.png" /></figure>

This is from the mps device:

<figure><img src="image_466.png" /></figure>
<figure><img src="image_435.png" /><figcaption>Figure 2: Feature model (excerpt). Inverse (resp. Relation-WithPi) corresponds to checking the property $(x*z)/(y*z) = x/y$ (resp. $(x*z*\pi)/(y*z*\pi) = x/y$) with $z, y \neq 0$</figcaption></figure>

```python
def equality_test(equality_check: EqualityCheck, x, y, z) -> bool:
    if equality_check == EqualityCheck.ASSOCIATIVITY:
        return x+(y+z) == (x+y)+z
    elif equality_check == EqualityCheck.MULT_INV:
        return (x * z) / (y * z) == x / y
    elif equality_check == EqualityCheck.MULT_INV_PI:
        return (x * z * math.pi) / (y * z * math.pi) == (x / y)
```

```python
fn check_ratio(config: &Config, x: f64, y: f64, z: f64) -> bool:
    if let Some.errorMargin = config.errorMargin {
        #[cfg(feature = "associativity")]
        ((x + y) + z - x - (y + z)).abs() < errorMargin
    }
    #[cfg(feature = "multInverse")]
    ((x * z) / (y * z) - x / y).abs() < errorMargin
}
#[cfg(feature = "multInverse_pi")]
((x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) - x / y).abs() < errorMargin
}

} else {
    #[cfg(feature = "associativity")]
    (x + y) + z == x + (y + z)
}

#[cfg(feature = "multInverse")]
{(x * z) / (y * z) == x / y
}

#[cfg(feature = "multInverse_pi")]
{(x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) == (x / y)
}
}
```
<figure><img src="image_467.png" /><figcaption>Figure 2: Feature model (excerpt). Inverse (resp. Relation-WithPi) corresponds to checking the property $(x*z)/(y*z) = x/y$ (resp. $(x*z*\pi)/(y*z*\pi) = x/y$) with $z, y \neq 0$</figcaption></figure>

```python
def equality_test(equality_check: EqualityCheck, x, y, z) -> bool:
    if equality_check == EqualityCheck.ASSOCIATIVITY:
        return x+(y+z) == (x+y)+z
    elif equality_check == EqualityCheck.MULT_INV:
        return (x * z) / (y * z) == x / y
    elif equality_check == EqualityCheck.MULT_INV_PI:
        return (x * z * math.pi) / (y * z * math.pi) == (x / y)
```

```python
fn check_ratio(config: &Config, x: f64, y: f64, z: f64) -> bool:
    if let Some errorMargin = config.errorMargin {
        #[cfg(feature = "associativity")]
        ((x + y) + z - x - (y + z)).abs() < errorMargin
    }
    #[cfg(feature = "multInverse")]
    ((x * z) / (y * z) - x / y).abs() < errorMargin
}
#[cfg(feature = "multInverse_pi")]
((x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) - x / y).abs() < errorMargin
}

} else {
    #[cfg(feature = "associativity")]
    (x + y) + z == x + (y + z)
}

#[cfg(feature = "multInverse")]
{(x * z) / (y * z) == x / y
}

#[cfg(feature = "multInverse_pi")]
{(x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) == (x / y)
}
}
```
<figure><img src="image_436.png" /><figcaption>Figure 2: Feature model (excerpt). Inverse (resp. Relation-WithPi) corresponds to checking the property $(x*z)/(y*z) = x/y$ (resp. $(x*z*\pi)/(y*z*\pi) = x/y$) with $z, y \neq 0$</figcaption></figure>

```c
fn associativity_test(config: &Config) -> bool {
    let mut rng = thread rng();
    // TODO: this variant for generating random
    // let x = rng.gen::<f64>();
    // let y = rng.gen::<f64>();
    // let z = rng.gen::<f64>();
    let x = rng.gen_range(0.000_000_000_000_001..100.0); // TODO: variation point for range min, max value
    let y = rng.gen_range(0.000_000_000_000_001..100.0);
    let z = rng.gen_range(0.000_000_000_000_001..100.0);
    check_ratio(config, x, y, z)
}

fn proportion(config: &Config, number: i32, seed_val: u64) -> i32 {
    StdRng::seed_from_u64(seed_val);
    let mut ok = 0;
    for _ in 0..number {
        if associativity_test(config) {
            ok += 1;
        }
    }
    ok * 100 / number
}
```

```c
if errorMargin:
    variability_misc = f"--errorMargin {errorMargin}"
    cmd_args = [
        "cargo",
        "run",
        "--features",
        feature,
        "-q",
        "--",
        "--errorMargin",
        errorMargin,
    ]
```

```c
fn check_ratio(config: &Config, x: f64, y: f64, z: f64) -> bool {
    if let Some errorMargin = config.errorMargin {
        @[cfg(feature = "associativity")]
        {
            [(x + y) + z * x * (y + z)].abs() < errorMargin
        }
        @[cfg(feature = "multInverse")]
        {
            [(x * z) / (y * z) - x / y].abs() < errorMargin
        }
        @[cfg(feature = "multInversePi")]
        {
            [(x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) - x / y].abs() < errorMargin
        }
    } else {
        @[cfg(feature = "associativity")]
        {
            (x + y) + z == x + (y + z)
        }
        @[cfg(feature = "multInverse")]
        {
            (x * z) / (y * z) == x / y
        }
        @[cfg(feature = "multInversePi")]
        {
            (x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) == (x / y)
        }
    }
}
```
<figure><img src="image_468.png" /><figcaption>Figure 2: Feature model (excerpt). Inverse (resp. Relation-WithPi) corresponds to checking the property $(x*z)/(y*z) = x/y$ (resp. $(x*z*\pi)/(y*z*\pi) = x/y$) with $z, y \neq 0$</figcaption></figure>

```c
fn associativity_test(config: &Config) -> bool {
    let mut rng = thread rng();
    // TODO: this variant for generating random
    // let x = rng.gen::<f64>();
    // let y = rng.gen::<f64>();
    // let z = rng.gen::<f64>();
    let x = rng.gen_range(0.000_000_000_000_001..100.0); // TODO: variation point for range min, max value
    let y = rng.gen_range(0.000_000_000_000_001..100.0);
    let z = rng.gen_range(0.000_000_000_000_001..100.0);
    check_ratio(config, x, y, z)
}

fn proportion(config: &Config, number: i32, seed_val: u64) -> i32 {
    StdRng::seed_from_u64(seed_val);
    let mut ok = 0;
    for _ in 0..number {
        if associativity_test(config) {
            ok += 1;
        }
    }
    ok * 100 / number
}
```

```c
if errorMargin:
    variability_misc = f"--errorMargin {errorMargin}"
    cmd_args = [
        "cargo",
        "run",
        "--features",
        feature,
        "-q",
        "--",
        "--errorMargin",
        errorMargin,
    ]
```

```c
fn check_ratio(config: &Config, x: f64, y: f64, z: f64) -> bool {
    if let Some errorMargin = config.errorMargin {
        @[cfg(feature = "associativity")]
        {
            [(x + y) + z * x * (y + z)].abs() < errorMargin
        }
        @[cfg(feature = "multInverse")]
        {
            [(x * z) / (y * z) - x / y].abs() < errorMargin
        }
        @[cfg(feature = "multInversePi")]
        {
            [(x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) - x / y].abs() < errorMargin
        }
    } else {
        @[cfg(feature = "associativity")]
        {
            (x + y) + z == x + (y + z)
        }
        @[cfg(feature = "multInverse")]
        {
            (x * z) / (y * z) == x / y
        }
        @[cfg(feature = "multInversePi")]
        {
            (x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) == (x / y)
        }
    }
}
```
<table><thead><tr><th>Language</th><th>Library</th><th>System</th><th>Compiler</th><th>VariabilityMisc</th><th>EqualityCheck</th><th>NumberGenerations</th><th>Repeat</th><th>min</th><th>max</th><th>std</th><th>mean</th></tr></thead><tbody><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV</td><td>100</td><td>10</td><td>60.0</td><td>71.0</td><td>3.562302626111375</td><td>65.1</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>51.0</td><td>63.0</td><td>3.330165161069343</td><td>55.9</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV</td><td>100</td><td>10</td><td>62.0</td><td>62.0</td><td>0.0</td><td>62.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>47.0</td><td>47.0</td><td>0.0</td><td>47.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>associativity</td><td>100</td><td>10</td><td>71.0</td><td>82.0</td><td>3.3466401061363023</td><td>76.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>mult-inverse</td><td>100</td><td>10</td><td>58.0</td><td>78.0</td><td>6.0</td><td>66.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>mult-inverse-pi</td><td>100</td><td>10</td><td>42.0</td><td>64.0</td><td>5.885575587824865</td><td>53.4</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>associativity</td><td>100</td><td>10</td><td>81.0</td><td>81.0</td><td>0.0</td><td>81.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse</td><td>100</td><td>10</td><td>70.0</td><td>70.0</td><td>0.0</td><td>70.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse-pi</td><td>100</td><td>10</td><td>56.0</td><td>56.0</td><td>0.0</td><td>56.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV</td><td>100</td><td>10</td><td>62.0</td><td>72.0</td><td>2.764054992217051</td><td>66.6</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>47.0</td><td>57.0</td><td>2.808914381037628</td><td>53.1</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV</td><td>100</td><td>10</td><td>67.0</td><td>67.0</td><td>0.0</td><td>67.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>53.0</td><td>53.0</td><td>0.0</td><td>53.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>74.0</td><td>90.0</td><td>4.6097722286464435</td><td>82.5</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>MULT_INV</td><td>100</td><td>10</td><td>60.0</td><td>79.0</td><td>6.16765757804371</td><td>68.6</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>49.0</td><td>59.0</td><td>2.8301943396169813</td><td>54.3</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>89.0</td><td>89.0</td><td>0.0</td><td>89.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_INV</td><td>100</td><td>10</td><td>73.0</td><td>73.0</td><td>0.0</td><td>73.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>55.0</td><td>55.0</td><td>0.0</td><td>55.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None approximate equality of Julia lang</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None approximate equality of Julia lang</td><td>MULT_INV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr></tbody></table>

https://github.com/FAMILIAR-project/reproducibility-associativity/
<table><thead><tr><th>Language</th><th>Library</th><th>System</th><th>Compiler</th><th>VariabilityMisc</th><th>EqualityCheck</th><th>NumberGenerations</th><th>Repeat</th><th>min</th><th>max</th><th>std</th><th>mean</th></tr></thead><tbody><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV</td><td>100</td><td>10</td><td>60.0</td><td>71.0</td><td>3.562302626111375</td><td>65.1</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>51.0</td><td>63.0</td><td>3.330165161069343</td><td>55.9</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV</td><td>100</td><td>10</td><td>62.0</td><td>62.0</td><td>0.0</td><td>62.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>47.0</td><td>47.0</td><td>0.0</td><td>47.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>associativity</td><td>100</td><td>10</td><td>71.0</td><td>82.0</td><td>3.3466401061363023</td><td>76.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>mult-inverse</td><td>100</td><td>10</td><td>58.0</td><td>78.0</td><td>6.0</td><td>66.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>mult-inverse-pi</td><td>100</td><td>10</td><td>42.0</td><td>64.0</td><td>5.885575587824865</td><td>53.4</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>associativity</td><td>100</td><td>10</td><td>81.0</td><td>81.0</td><td>0.0</td><td>81.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse</td><td>100</td><td>10</td><td>70.0</td><td>70.0</td><td>0.0</td><td>70.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse-pi</td><td>100</td><td>10</td><td>56.0</td><td>56.0</td><td>0.0</td><td>56.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV</td><td>100</td><td>10</td><td>62.0</td><td>72.0</td><td>2.764054992217051</td><td>66.6</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>47.0</td><td>57.0</td><td>2.808914381037628</td><td>53.1</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV</td><td>100</td><td>10</td><td>67.0</td><td>67.0</td><td>0.0</td><td>67.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>53.0</td><td>53.0</td><td>0.0</td><td>53.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>74.0</td><td>90.0</td><td>4.6097722286464435</td><td>82.5</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>MULT_INV</td><td>100</td><td>10</td><td>60.0</td><td>79.0</td><td>6.16765757804371</td><td>68.6</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>49.0</td><td>59.0</td><td>2.8301943396169813</td><td>54.3</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>89.0</td><td>89.0</td><td>0.0</td><td>89.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_INV</td><td>100</td><td>10</td><td>73.0</td><td>73.0</td><td>0.0</td><td>73.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>55.0</td><td>55.0</td><td>0.0</td><td>55.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None approximate equality of Julia lang</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None approximate equality of Julia lang</td><td>MULT_INV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr></tbody></table>

https://github.com/FAMILIAR-project/reproducibility-associativity/