Repository: ytdl-org/youtube-dl

Issue:
On Windows, "ERROR: Cannot write metadata to JSON file" if .json file already exists
If the .json file already exists, the download fails with "ERROR: Cannot write metadata to JSON file", even when using `--ignore-errors`.  This happens on Windows 7 and 8.1 with Python 2.7.8 or Python 3.4.1.  Below I tested with Python 3.4.1 on Windows 8.1, running 480b7c32a94db050bb216519d3b7fd4589d1e8fb:

```
C:\Users\root\youtube-dl>C:\Python34\python.exe -m youtube_dl --write-info-json "d9Qf_yFKawI"
[youtube] Confirming age
[youtube] d9Qf_yFKawI: Downloading webpage
[youtube] d9Qf_yFKawI: Downloading video info webpage
[youtube] d9Qf_yFKawI: Extracting video information
[youtube] d9Qf_yFKawI: Downloading DASH manifest
[info] Writing video description metadata as JSON to: Shokojo Seira ?????? Eps 8 ENG-d9Qf_yFKawI.info.json
[download] Destination: Shokojo Seira ?????? Eps 8 ENG-d9Qf_yFKawI.mp4
[download]   1.1% of 590.96MiB at  2.57MiB/s ETA 03:47
ERROR: Interrupted by user

C:\Users\root\youtube-dl>C:\Python34\python.exe -m youtube_dl --write-info-json "d9Qf_yFKawI"
[youtube] Confirming age
[youtube] d9Qf_yFKawI: Downloading webpage
[youtube] d9Qf_yFKawI: Downloading video info webpage
[youtube] d9Qf_yFKawI: Extracting video information
[youtube] d9Qf_yFKawI: Downloading DASH manifest
[info] Writing video description metadata as JSON to: Shokojo Seira ?????? Eps 8 ENG-d9Qf_yFKawI.info.json
ERROR: Cannot write metadata to JSON file Shokojo Seira ?????? Eps 8 ENG-d9Qf_yFKawI.info.json

C:\Users\root\youtube-dl>C:\Python34\python.exe -m youtube_dl --write-info-json "Q-bmvMWqWmY"
[youtube] Confirming age
[youtube] Q-bmvMWqWmY: Downloading webpage
[youtube] Q-bmvMWqWmY: Downloading video info webpage
[youtube] Q-bmvMWqWmY: Extracting video information
[youtube] Q-bmvMWqWmY: Downloading DASH manifest
[info] Writing video description metadata as JSON to: advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json
[download] Destination: advanced scratch - Emacs Bites-Q-bmvMWqWmY.mp4
[download]  12.3% of 8.15MiB at  2.10MiB/s ETA 00:03
ERROR: Interrupted by user

C:\Users\root\youtube-dl>C:\Python34\python.exe -m youtube_dl --write-info-json "Q-bmvMWqWmY"
[youtube] Confirming age
[youtube] Q-bmvMWqWmY: Downloading webpage
[youtube] Q-bmvMWqWmY: Downloading video info webpage
[youtube] Q-bmvMWqWmY: Extracting video information
[youtube] Q-bmvMWqWmY: Downloading DASH manifest
[info] Writing video description metadata as JSON to: advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json
ERROR: Cannot write metadata to JSON file advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json

C:\Users\root\youtube-dl>C:\Python34\python.exe -m youtube_dl --write-info-json --ignore-errors "Q-bmvMWqWmY"
[youtube] Confirming age
[youtube] Q-bmvMWqWmY: Downloading webpage
[youtube] Q-bmvMWqWmY: Downloading video info webpage
[youtube] Q-bmvMWqWmY: Extracting video information
[youtube] Q-bmvMWqWmY: Downloading DASH manifest
[info] Writing video description metadata as JSON to: advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json
ERROR: Cannot write metadata to JSON file advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json
```

I think this regressed a few months ago.

On Windows, "ERROR: Cannot write metadata to JSON file" if .json file already exists
If the .json file already exists, the download fails with "ERROR: Cannot write metadata to JSON file", even when using `--ignore-errors`.  This happens on Windows 7 and 8.1 with Python 2.7.8 or Python 3.4.1.  Below I tested with Python 3.4.1 on Windows 8.1, running 480b7c32a94db050bb216519d3b7fd4589d1e8fb:

```
C:\Users\root\youtube-dl>C:\Python34\python.exe -m youtube_dl --write-info-json "d9Qf_yFKawI"
[youtube] Confirming age
[youtube] d9Qf_yFKawI: Downloading webpage
[youtube] d9Qf_yFKawI: Downloading video info webpage
[youtube] d9Qf_yFKawI: Extracting video information
[youtube] d9Qf_yFKawI: Downloading DASH manifest
[info] Writing video description metadata as JSON to: Shokojo Seira ?????? Eps 8 ENG-d9Qf_yFKawI.info.json
[download] Destination: Shokojo Seira ?????? Eps 8 ENG-d9Qf_yFKawI.mp4
[download]   1.1% of 590.96MiB at  2.57MiB/s ETA 03:47
ERROR: Interrupted by user

C:\Users\root\youtube-dl>C:\Python34\python.exe -m youtube_dl --write-info-json "d9Qf_yFKawI"
[youtube] Confirming age
[youtube] d9Qf_yFKawI: Downloading webpage
[youtube] d9Qf_yFKawI: Downloading video info webpage
[youtube] d9Qf_yFKawI: Extracting video information
[youtube] d9Qf_yFKawI: Downloading DASH manifest
[info] Writing video description metadata as JSON to: Shokojo Seira ?????? Eps 8 ENG-d9Qf_yFKawI.info.json
ERROR: Cannot write metadata to JSON file Shokojo Seira ?????? Eps 8 ENG-d9Qf_yFKawI.info.json

C:\Users\root\youtube-dl>C:\Python34\python.exe -m youtube_dl --write-info-json "Q-bmvMWqWmY"
[youtube] Confirming age
[youtube] Q-bmvMWqWmY: Downloading webpage
[youtube] Q-bmvMWqWmY: Downloading video info webpage
[youtube] Q-bmvMWqWmY: Extracting video information
[youtube] Q-bmvMWqWmY: Downloading DASH manifest
[info] Writing video description metadata as JSON to: advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json
[download] Destination: advanced scratch - Emacs Bites-Q-bmvMWqWmY.mp4
[download]  12.3% of 8.15MiB at  2.10MiB/s ETA 00:03
ERROR: Interrupted by user

C:\Users\root\youtube-dl>C:\Python34\python.exe -m youtube_dl --write-info-json "Q-bmvMWqWmY"
[youtube] Confirming age
[youtube] Q-bmvMWqWmY: Downloading webpage
[youtube] Q-bmvMWqWmY: Downloading video info webpage
[youtube] Q-bmvMWqWmY: Extracting video information
[youtube] Q-bmvMWqWmY: Downloading DASH manifest
[info] Writing video description metadata as JSON to: advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json
ERROR: Cannot write metadata to JSON file advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json

C:\Users\root\youtube-dl>C:\Python34\python.exe -m youtube_dl --write-info-json --ignore-errors "Q-bmvMWqWmY"
[youtube] Confirming age
[youtube] Q-bmvMWqWmY: Downloading webpage
[youtube] Q-bmvMWqWmY: Downloading video info webpage
[youtube] Q-bmvMWqWmY: Extracting video information
[youtube] Q-bmvMWqWmY: Downloading DASH manifest
[info] Writing video description metadata as JSON to: advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json
ERROR: Cannot write metadata to JSON file advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json
```

I think this regressed a few months ago.



Discussion:
Patching `YoutubeDL.py` to raise the error with:

``` python
                try:
                    write_json_file(info_dict, infofn)
                except (OSError, IOError):
                    raise
                    self.report_error('Cannot write metadata to JSON file ' + infofn)
                    return
```

and testing again shows:

```
# PYTHONPATH=. p3 -m youtube_dl --write-info-json Q-bmvMWqWmY
[youtube] Confirming age
[youtube] Q-bmvMWqWmY: Downloading webpage
[youtube] Q-bmvMWqWmY: Downloading video info webpage
[youtube] Q-bmvMWqWmY: Extracting video information
[youtube] Q-bmvMWqWmY: Downloading DASH manifest
[info] Writing video description metadata as JSON to: advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json
Traceback (most recent call last):
  File "C:\Python34-64\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python34-64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\__main__.py", line 18, in <module>
    youtube_dl.main()
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\__init__.py", line 348, in main
    _real_main(argv)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\__init__.py", line 338, in _real_main
    retcode = ydl.download(all_urls)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\YoutubeDL.py", line 1095, in download
    res = self.extract_info(url)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\YoutubeDL.py", line 547, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\YoutubeDL.py", line 593, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\YoutubeDL.py", line 870, in process_video_result
    self.process_info(new_info)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\YoutubeDL.py", line 1004, in process_info
    write_json_file(info_dict, infofn)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\utils.py", line 112, in write_json_file
    os.rename(tf.name, fn)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\root\\Documents\\ExpClj\\youtube-dl\\advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json.gyq5n8bs.tmp' -> 'advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json'
```

Patching `YoutubeDL.py` to raise the error with:

``` python
                try:
                    write_json_file(info_dict, infofn)
                except (OSError, IOError):
                    raise
                    self.report_error('Cannot write metadata to JSON file ' + infofn)
                    return
```

and testing again shows:

```
# PYTHONPATH=. p3 -m youtube_dl --write-info-json Q-bmvMWqWmY
[youtube] Confirming age
[youtube] Q-bmvMWqWmY: Downloading webpage
[youtube] Q-bmvMWqWmY: Downloading video info webpage
[youtube] Q-bmvMWqWmY: Extracting video information
[youtube] Q-bmvMWqWmY: Downloading DASH manifest
[info] Writing video description metadata as JSON to: advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json
Traceback (most recent call last):
  File "C:\Python34-64\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python34-64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\__main__.py", line 18, in <module>
    youtube_dl.main()
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\__init__.py", line 348, in main
    _real_main(argv)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\__init__.py", line 338, in _real_main
    retcode = ydl.download(all_urls)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\YoutubeDL.py", line 1095, in download
    res = self.extract_info(url)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\YoutubeDL.py", line 547, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\YoutubeDL.py", line 593, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\YoutubeDL.py", line 870, in process_video_result
    self.process_info(new_info)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\YoutubeDL.py", line 1004, in process_info
    write_json_file(info_dict, infofn)
  File "C:\Users\root\Documents\ExpClj\youtube-dl\youtube_dl\utils.py", line 112, in write_json_file
    os.rename(tf.name, fn)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\root\\Documents\\ExpClj\\youtube-dl\\advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json.gyq5n8bs.tmp' -> 'advanced scratch - Emacs Bites-Q-bmvMWqWmY.info.json'
```


Resolving patch:
diff --git a/youtube_dl/utils.py b/youtube_dl/utils.py
--- a/youtube_dl/utils.py
+++ b/youtube_dl/utils.py
@@ -71,10 +71,10 @@ def preferredencoding():
 
 
 def write_json_file(obj, fn):
-    """ Encode obj as JSON and write it to fn, atomically """
+    """ Encode obj as JSON and write it to fn, atomically if possible """
 
     fn = encodeFilename(fn)
-    if sys.version_info < (3, 0):
+    if sys.version_info < (3, 0) and sys.platform != 'win32':
         encoding = get_filesystem_encoding()
         # os.path.basename returns a bytes object, but NamedTemporaryFile
         # will fail if the filename contains non ascii characters unless we
@@ -108,6 +108,13 @@ def write_json_file(obj, fn):
     try:
         with tf:
             json.dump(obj, tf)
+        if sys.platform == 'win32':
+            # Need to remove existing file on Windows, else os.rename raises
+            # WindowsError or FileExistsError.
+            try:
+                os.unlink(fn)
+            except OSError:
+                pass
         os.rename(tf.name, fn)
     except:
         try:


Test patch: