Repository: TheLastGimbus/GooglePhotosTakeoutHelper (language: python)

Issue:
Unit Tests
First of all thank you very much for the great work so far! 🚀

I've got a huge collection of Google Photos which I'd like to pipe through the takeout helper. Initial tests revealed that my takeout contains many edge cases which the helper does not handle correctly yet. I'd be super happy to contribute, but I'd afraid to break existing functionality.

Are there any plans to bring some high level unit test structure into place? Maybe at a full scale integration level first? Something simple like a set of predefined input and reference output folders?

Doesn't even need to be automated in a CI, local execution would already be a win I think.


Discussion / hints:
Oh, jeez...

Oh, yes yes, you are very welcome here! I just really didn't ever expect this *simple script* to grow to any size like that...

> plans to bring some high level unit test structure into place

I may repeat myself like 3rd time in those comments, but this script is really shitty to be honest. If I would write it righ now, nowing exactly what folder structure/edge cases there are, and *how many people* will use it, I would do this differently:
- for now, it kinda "walks through" the files and does stuff with them
- what I *would do*, is to generate some data strucutres from *all* input files and data, and then play with it

Tests? Yeah, we could write some of those :+1: Righ now, I'm just in the middle of writing a GitHub action that runs the script 30 times on sample folder and measures the time. Perhaps some way to quickly test individual functions, like `find_json_for_file`.

> my takeout contains many edge cases

Could you name some? I already have a routine of "people give me their weird photos, I ~~tell them to send me a donation~~ add it in some PR. Maybe the last time you checked it was some old version - we have a lot of improvments now that protect from edge cases :rocket: 

>  I'd be super happy to contribute

You can wait till I merge those #62 and #63 and maybe I will add some documenation after that

I need to stop lying to myself that this project is done and finished, because it never will be xD - I will add some `contributing.mg` also
Yes, you can now watch me have fun at #66
> Perhaps some way to quickly test individual functions, like `find_json_for_file`.

Fine grained tests on a function level would "manifest" the current structure and get in the way of refactorings like the data structure one you mentioned.

> I already have a routine of "people give me their weird photos

Correct me if I'm wrong, but we actually only need file _names_, not the actual contents, right? Or do we also sometimes need the JSON file contents? (to be honest I haven't read the code in detail yet)

> I need to stop lying to myself that this project is done and finished, because it never will be xD

I guess that's a given with Google as a moving target on the other side ^^

I'll try to find time this week to open a draft PR to sketch out a first basic test setup proposal.
> we actually only need file names

Oh no no no, we use *everything* to get right photo data

I will make some general "what this script does" description in `contributing.md` - I can ping you when I'm done, so you won't waste your patiance reverse-engineering code

> first basic test setup proposal

Currently, everything is like:
```python
def main():
  varib = 0
  def other_fun():  # Functions inside main() function
    nonlocal varib  # Ton of global variables <3
```
So I don't know whtether that's really test-able, but I started to think about some general tests that would go like:
 - take folder with pngs (they don't have exifs), but with jsons
 - assert that they have correct dates
 - take folder with photos which' locations were edited in Google Photos editor
 - assert that their output exifs have this location
 - etc

If you want to help:
1. Drop me your edge-case photos - maybe we can do something with them :+1:
2. Also you can give me some exaple photos with stuff to assert - weird formats, stuff where you edited location, duplicates etc

> Oh no no no, we use everything to get right photo data

Everything being EXIF, the file JSON, the folder/album JSON aaand..? (btw, potential last resort fallback: the filename). The point I'm getting at is that we probably don't need the actual image data, like "the pixels". At least I, and I can imagine more people, will be more comfortable sharing their photos which will eventually be hosted in this repo if they don't contain any more potential private information than the filename, EXIF data and the JSON metadata (which could also be cleaned manually if necessary).

> So I don't know whtether that's really test-able, but I started to think about some general tests that would go like:

Yep, that's pretty much what I had in mind. See #68 

> If you want to help:
> 1. Drop me your edge-case photos - maybe we can do something with them 👍
> 2. Also you can give me some exaple photos with stuff to assert - weird formats, stuff where you edited location, duplicates etc

Will do in #68 👍 

Fix patch:
diff --git a/.gitignore b/.gitignore
index f4b0e255..ac18a282 100644
--- a/.gitignore
+++ b/.gitignore
@@ -9,4 +9,5 @@ dist/
 *.egg-info
 .eggs/
 ALL_PHOTOS/
+output/
 *.log


Test patch:
diff --git a/scripts/init-for-tests b/scripts/init-for-tests
new file mode 100755
index 00000000..ad8a4e1c
--- /dev/null
+++ b/scripts/init-for-tests
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+# We go to GPTH/scripts/../../(parent folder)
+cd "$(dirname "$0")"
+cd ../..
+echo "Working in parent folder $(pwd)"
+
+# Download test repo at certain tag
+# Change this tag in future when test repo changes
+echo "Cloning test data repo..."
+git clone --depth 1 --branch v0.0.1 git@github.com:TheLastGimbus/__takeout_helper_test_data__.git
+
+# Now, run GPTH on it!
diff --git a/scripts/reset-test-data b/scripts/reset-test-data
new file mode 100755
index 00000000..16a7a605
--- /dev/null
+++ b/scripts/reset-test-data
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+# We go to GPTH/scripts/../../(parent folder is here)/test_data
+cd "$(dirname "$0")"
+cd ../../__takeout_helper_test_data__
+echo "Working in parent folder $(pwd)"
+
+# Remove all outputs and restore input files to original state
+rm -r output/
+git reset --hard
+
+echo "Test data folder is fresh and clean!"
diff --git a/scripts/run-gpth-on-test-data b/scripts/run-gpth-on-test-data
new file mode 100755
index 00000000..88080aac
--- /dev/null
+++ b/scripts/run-gpth-on-test-data
@@ -0,0 +1,15 @@
+#!/bin/bash
+# I know, this script is kinda boilerplate but it will look nicer in CI
+# Go to parent folder
+cd "$(dirname "$0")"
+cd ../..
+echo "Working in parent folder $(pwd)"
+
+# Now, run GPTH on it:
+echo "Running GPTH on test data repo"
+python3 GooglePhotosTakeoutHelper/google_photos_takeout_helper/__main__.py \
+  -i "__takeout_helper_test_data__/Google Photos" \
+  -o "__takeout_helper_test_data__/output"
+
+echo "=========================="
+echo "Now you can go to GPTH folder and run pytest :)"
diff --git a/tests/test_file_modified.py b/tests/test_file_modified.py
new file mode 100644
index 00000000..c5676a48
--- /dev/null
+++ b/tests/test_file_modified.py
@@ -0,0 +1,14 @@
+import json
+import pathlib
+
+test_assets_path = pathlib.Path('../__takeout_helper_test_data__')
+out_path = test_assets_path / 'output'
+asserts_path = test_assets_path / 'asserts'
+
+with open(asserts_path / 'file_modified.json', 'r') as f:
+    _data = json.load(f)
+    pairs = {**_data['photos'], **_data['videos']}
+
+for key in pairs:
+    file = out_path / key
+    assert file.stat().st_mtime == pairs[key]