File size: 17,119 Bytes
ef5fb31
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
2022-01-30 17:47:50,131 INFO    MainThread:19599 [internal.py:wandb_internal():87] W&B internal server running at pid: 19599, started at: 2022-01-30 17:47:50.130774
2022-01-30 17:47:50,135 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: check_version
2022-01-30 17:47:50,135 INFO    WriterThread:19599 [datastore.py:open_for_write():77] open: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/run-kzuygp3o.wandb
2022-01-30 17:47:50,139 DEBUG   SenderThread:19599 [sender.py:send():234] send: header
2022-01-30 17:47:50,139 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: check_version
2022-01-30 17:47:50,215 DEBUG   SenderThread:19599 [sender.py:send():234] send: run
2022-01-30 17:47:50,957 INFO    SenderThread:19599 [dir_watcher.py:__init__():169] watching files in: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files
2022-01-30 17:47:50,957 INFO    SenderThread:19599 [sender.py:_start_run_threads():804] run started: kzuygp3o with start time 1643564869
2022-01-30 17:47:50,958 DEBUG   SenderThread:19599 [sender.py:send():234] send: summary
2022-01-30 17:47:50,958 INFO    SenderThread:19599 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:47:50,959 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: run_start
2022-01-30 17:47:50,967 DEBUG   HandlerThread:19599 [meta.py:__init__():40] meta init
2022-01-30 17:47:50,967 DEBUG   HandlerThread:19599 [meta.py:__init__():54] meta init done
2022-01-30 17:47:50,968 DEBUG   HandlerThread:19599 [meta.py:probe():214] probe
2022-01-30 17:47:50,976 DEBUG   HandlerThread:19599 [meta.py:_setup_git():204] setup git
2022-01-30 17:47:51,010 DEBUG   HandlerThread:19599 [meta.py:_setup_git():211] setup git done
2022-01-30 17:47:51,010 DEBUG   HandlerThread:19599 [meta.py:_save_pip():58] save pip
2022-01-30 17:47:51,011 DEBUG   HandlerThread:19599 [meta.py:_save_pip():72] save pip done
2022-01-30 17:47:51,011 DEBUG   HandlerThread:19599 [meta.py:_save_conda():79] save conda
2022-01-30 17:47:51,503 DEBUG   HandlerThread:19599 [meta.py:_save_conda():89] save conda done
2022-01-30 17:47:51,503 DEBUG   HandlerThread:19599 [meta.py:probe():252] probe done
2022-01-30 17:47:51,511 DEBUG   SenderThread:19599 [sender.py:send():234] send: files
2022-01-30 17:47:51,511 INFO    SenderThread:19599 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-01-30 17:47:51,523 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: stop_status
2022-01-30 17:47:51,524 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: stop_status
2022-01-30 17:47:51,704 DEBUG   SenderThread:19599 [sender.py:send():234] send: config
2022-01-30 17:47:51,706 DEBUG   SenderThread:19599 [sender.py:send():234] send: metric
2022-01-30 17:47:51,707 DEBUG   SenderThread:19599 [sender.py:send():234] send: metric
2022-01-30 17:47:51,707 WARNING SenderThread:19599 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen)
2022-01-30 17:47:51,961 INFO    Thread-8  :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-metadata.json
2022-01-30 17:47:51,961 INFO    Thread-8  :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/requirements.txt
2022-01-30 17:47:51,961 INFO    Thread-8  :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log
2022-01-30 17:47:51,962 INFO    Thread-8  :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json
2022-01-30 17:47:51,962 INFO    Thread-8  :19599 [dir_watcher.py:_on_file_created():217] file/dir created: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/conda-environment.yaml
2022-01-30 17:47:52,037 INFO    Thread-11 :19599 [upload_job.py:push():137] Uploaded file /tmp/tmpl83m3xpuwandb/2em52v7y-wandb-metadata.json
2022-01-30 17:47:53,959 INFO    Thread-8  :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log
2022-01-30 17:47:57,751 DEBUG   SenderThread:19599 [sender.py:send():234] send: telemetry
2022-01-30 17:47:57,752 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:57,752 DEBUG   SenderThread:19599 [sender.py:send():234] send: exit
2022-01-30 17:47:57,753 INFO    SenderThread:19599 [sender.py:send_exit():366] handling exit code: 1
2022-01-30 17:47:57,753 INFO    SenderThread:19599 [sender.py:send_exit():368] handling runtime: 6
2022-01-30 17:47:57,754 INFO    SenderThread:19599 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:47:57,754 INFO    SenderThread:19599 [sender.py:send_exit():374] send defer
2022-01-30 17:47:57,754 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:57,755 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:57,755 INFO    HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 0
2022-01-30 17:47:57,756 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:57,756 INFO    SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 0
2022-01-30 17:47:57,756 INFO    SenderThread:19599 [sender.py:transition_state():387] send defer: 1
2022-01-30 17:47:57,757 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:57,757 INFO    HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 1
2022-01-30 17:47:57,792 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:57,793 INFO    SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 1
2022-01-30 17:47:57,793 INFO    SenderThread:19599 [sender.py:transition_state():387] send defer: 2
2022-01-30 17:47:57,794 DEBUG   SenderThread:19599 [sender.py:send():234] send: stats
2022-01-30 17:47:57,795 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:57,795 INFO    HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 2
2022-01-30 17:47:57,795 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:57,796 INFO    SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 2
2022-01-30 17:47:57,796 INFO    SenderThread:19599 [sender.py:transition_state():387] send defer: 3
2022-01-30 17:47:57,796 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:57,797 INFO    HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 3
2022-01-30 17:47:57,797 DEBUG   SenderThread:19599 [sender.py:send():234] send: summary
2022-01-30 17:47:57,798 INFO    SenderThread:19599 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-30 17:47:57,798 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:57,798 INFO    SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 3
2022-01-30 17:47:57,798 INFO    SenderThread:19599 [sender.py:transition_state():387] send defer: 4
2022-01-30 17:47:57,799 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:57,799 INFO    HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 4
2022-01-30 17:47:57,799 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:57,799 INFO    SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 4
2022-01-30 17:47:57,857 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:57,962 INFO    Thread-8  :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json
2022-01-30 17:47:57,962 INFO    Thread-8  :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log
2022-01-30 17:47:58,077 INFO    SenderThread:19599 [sender.py:transition_state():387] send defer: 5
2022-01-30 17:47:58,077 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:58,078 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:58,079 INFO    HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 5
2022-01-30 17:47:58,079 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:58,079 INFO    SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 5
2022-01-30 17:47:58,079 INFO    SenderThread:19599 [dir_watcher.py:finish():283] shutting down directory watcher
2022-01-30 17:47:58,180 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:58,962 INFO    Thread-8  :19599 [dir_watcher.py:_on_file_modified():230] file/dir modified: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/config.yaml
2022-01-30 17:47:58,964 INFO    SenderThread:19599 [dir_watcher.py:finish():313] scan: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files
2022-01-30 17:47:58,965 INFO    SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log output.log
2022-01-30 17:47:58,965 INFO    SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-metadata.json wandb-metadata.json
2022-01-30 17:47:58,966 INFO    SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/config.yaml config.yaml
2022-01-30 17:47:58,970 INFO    SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/conda-environment.yaml conda-environment.yaml
2022-01-30 17:47:58,977 INFO    SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json wandb-summary.json
2022-01-30 17:47:58,978 INFO    SenderThread:19599 [dir_watcher.py:finish():327] scan save: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/requirements.txt requirements.txt
2022-01-30 17:47:58,985 INFO    SenderThread:19599 [sender.py:transition_state():387] send defer: 6
2022-01-30 17:47:58,986 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:58,986 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:58,987 INFO    HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 6
2022-01-30 17:47:58,988 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:58,993 INFO    SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 6
2022-01-30 17:47:58,994 INFO    SenderThread:19599 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 17:47:59,089 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,089 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,192 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,192 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,295 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,296 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,399 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,399 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,490 INFO    Thread-14 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/wandb-summary.json
2022-01-30 17:47:59,495 INFO    Thread-15 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/requirements.txt
2022-01-30 17:47:59,502 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,502 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,513 INFO    Thread-12 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/output.log
2022-01-30 17:47:59,591 INFO    Thread-13 :19599 [upload_job.py:push():137] Uploaded file /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/files/config.yaml
2022-01-30 17:47:59,604 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,605 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,707 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:47:59,707 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:47:59,793 INFO    Thread-7  :19599 [sender.py:transition_state():387] send defer: 7
2022-01-30 17:47:59,794 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:47:59,794 INFO    HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 7
2022-01-30 17:47:59,794 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:47:59,794 INFO    SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 7
2022-01-30 17:47:59,809 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:48:00,234 INFO    SenderThread:19599 [sender.py:transition_state():387] send defer: 8
2022-01-30 17:48:00,235 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:48:00,236 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:48:00,236 INFO    HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 8
2022-01-30 17:48:00,237 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:48:00,237 INFO    SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 8
2022-01-30 17:48:00,237 INFO    SenderThread:19599 [sender.py:transition_state():387] send defer: 9
2022-01-30 17:48:00,238 DEBUG   SenderThread:19599 [sender.py:send():234] send: final
2022-01-30 17:48:00,238 DEBUG   SenderThread:19599 [sender.py:send():234] send: footer
2022-01-30 17:48:00,239 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: defer
2022-01-30 17:48:00,239 INFO    HandlerThread:19599 [handler.py:handle_request_defer():147] handle defer: 9
2022-01-30 17:48:00,239 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: defer
2022-01-30 17:48:00,239 INFO    SenderThread:19599 [sender.py:send_request_defer():383] handle sender defer: 9
2022-01-30 17:48:00,338 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-30 17:48:00,338 DEBUG   SenderThread:19599 [sender.py:send_request():248] send_request: poll_exit
2022-01-30 17:48:00,338 INFO    SenderThread:19599 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 17:48:00,657 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: get_summary
2022-01-30 17:48:00,659 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: sampled_history
2022-01-30 17:48:00,660 DEBUG   HandlerThread:19599 [handler.py:handle_request():130] handle_request: shutdown
2022-01-30 17:48:00,660 INFO    HandlerThread:19599 [handler.py:finish():731] shutting down handler
2022-01-30 17:48:01,238 INFO    WriterThread:19599 [datastore.py:close():281] close: /workspace/xls-r-300m-fr/wandb/run-20220130_174749-kzuygp3o/run-kzuygp3o.wandb
2022-01-30 17:48:01,655 INFO    SenderThread:19599 [sender.py:finish():1070] shutting down sender
2022-01-30 17:48:01,655 INFO    SenderThread:19599 [file_pusher.py:finish():177] shutting down file pusher
2022-01-30 17:48:01,655 INFO    SenderThread:19599 [file_pusher.py:join():182] waiting for file pusher
2022-01-30 17:48:01,659 INFO    MainThread:19599 [internal.py:handle_exit():77] Internal process exited