Package: google-takeout-parser
google_takeout_parserParses data out of yourGoogle Takeout(History, Activity, Youtube, Locations, etc...). This:parses both the Historical HTML and new JSON format for Google Takeoutscaches individual takeout results behindcachewmerge multiple takeouts into unique eventsInstallationUsageCLI UsageLibrary UsageLegacy HTML ParsingContributingTestingThis doesn't handle all cases, but I have yet to find a parser that does, so here is my attempt at parsing what I see as the most useful info from it. The Google Takeout is pretty particular, and the contents of the directory depend on what you select while exporting. Unhandled files will warn, though feel free toPR a parserorcreate an issueif this doesn't parse some part you want.This can take a few minutes to parse depending on what you have in your Takeout (especially while using the old HTML format), so this usescachewto cache the function result for each Takeout you may have. That means this'll take a few minutes the first time parsing a takeout, but then only a few seconds every subsequent time.Since the Takeout slowly removes old events over time, I would recommend periodically (personally I do it once every few months) backing up your data, to not lose any old events and get data from new ones. To use, go totakeout.google.com; For Reference, once on that page, I hitDeselect All, then select:ChromeGoogle Play StoreLocation HistorySelect JSON as formatMy ActivitySelect JSON as formatYoutube and Youtube MusicSelect JSON as formatIn options, deselectmusic-library-songs,music-uploadsandvideosBe sure to select JSON instead of HTML whenever possible. Code to parse the HTML format is included here, but it is treated as legacy code and comes with worse performance and a myriad of other issues. Seelegacy html parsingThe process for getting these isn't that great -- you have to manually go totakeout.google.comevery few months, select what you want to export info for, and then it puts the zipped file into your google drive. You can tell it to run it at specific intervals, but I personally haven't found that to be that reliable.This currently parses:Activity (from dozens of Google Services) -My Activity/*.html|*.json)Chrome History -Chrome/BrowserHistory.jsonGoogle Play Installs -Google Play Store/Installs.jsonLocation History:Semantic Location HistoryLocation History/Semantic Location History/*Location HistoryLocation History/Location History.json,Location History/Records.jsonYoutube:History -YouTube and YouTube Music/history/*.html|*.jsonCommentsLegacy HTML Comment format:YouTube and YouTube Music/my-comments/*.htmlNew CSV/JSON format (mostly CSV, but the comment contents itself are a JSON blob):Youtube/comments/comments.csvYoutube/live chats/live chats.csvLive Chat Messages -YouTube and YouTube Music/my-live-chat-messages/*.htmlLikes:YouTube and YouTube Music/playlists/likes.jsonThis was extracted out ofmy HPImodules, which was in turn modified from the google files inkarlicoss/HPIInstallationRequirespython3.8+To install with pip, run:pip install google-takeout-parserUsageThe directory structure of the google takeout changes depending on your Google accounts main language. If this doesn't support your language, seecontributing. This currently supports:EN: EnglishDE: German (thanks to@parthux1)CLI UsageCan be accessed by eithergoogle_takeout_parserorpython -m google_takeout_parser. Offers a basic interface to list/clear the cache directory, and/or parse/merge a takeout and interact with it in a REPL:Usage: google_takeout_parser parse [OPTIONS] TAKEOUT_DIR

  Parse a takeout directory takeout

Options:
  -f, --filter [Activity|LikedYoutubeVideo|PlayStoreAppInstall|Location|ChromeHistory|YoutubeComment|PlaceVisit]
                                  Filter to only show events of this type
  -l, --locale [EN|DE]            Locale to use for matching filenames [default: EN]  [env var:
                                  GOOGLE_TAKEOUT_PARSER_LOCALE]
  -a, --action [repl|summary|json]
                                  What to do with the parsed result  [default: repl]
  --cache / --no-cache            [default: no-cache]
  -h, --help                      Show this message and exit.If you use a language this doesn't support, seecontributing.To clear thecachewcache:google_takeout_parser cache_dir clearA few examples of parsing takeouts:$ google_takeout_parser --quiet parse ~/data/Unpacked_Takout --cache
Interact with the export using res

In [1]: res[-2]
Out[1]: PlayStoreAppInstall(title='Hangouts', device_name='motorola moto g(7) play', dt=datetime.datetime(2020, 8, 2, 15, 51, 50, 180000, tzinfo=datetime.timezone.utc))

In [2]: len(res)
Out[2]: 236654$ google_takeout_parser --quiet merge ./Takeout-Old ./Takeout-New --action summary --no-cacheCounter({'Activity':366292,'Location':147581,'YoutubeComment':131,'PlayStoreAppInstall':122,'LikedYoutubeVideo':100,'ChromeHistory':4})Can also dump the info to JSON; e.g. to filter YouTube-related stuff from your Activity usingjq:google_takeout_parser--quietparse-ajson-fActivity--no-cache./Takeout-New|# select stuff like Youtube, m.youtube.com, youtube.com using jqjq'.[] | select(.header | ascii_downcase | test("youtube"))'|# grab the titleUrl, ignoring nullsjq'select(.titleUrl) | .titleUrl'-rAlso contains a small utility command to help move/extract the google takeout:$google_takeout_parsermove--from~/Downloads/takeout*.zip--to-dir~/data/google_takeout--extract
Extracting/home/sean/Downloads/takeout-20211023T070558Z-001.zipto/tmp/tmp07ua_0id
Moving/tmp/tmp07ua_0id/Takeoutto/home/sean/data/google_takeout/Takeout-1634993897
$ls-1~/data/google_takeout/Takeout-1634993897
archive_browser.html
Chrome'Google Play Store''Location History''My Activity''YouTube and YouTube Music'Library UsageAssuming you maintain an unpacked view, e.g. like:$ tree -L 1 ./Takeout-1599315526
./Takeout-1599315526
├── Google Play Store
├── Location History
├── My Activity
└── YouTube and YouTube MusicTo parse one takeout:fromgoogle_takeout.path_dispatchimportTakeoutParsertp=TakeoutParser("/full/path/to/Takeout-1599315526")# to check if files are all handledtp.dispatch_map()# to parse without caching the results in ~/.cache/google_takeout_parseruncached=list(tp.parse())# to parse with cachew cache https://github.com/karlicoss/cachewcached=list(tp.parse(cache=True))To parse a locale this doesn't support yet, you can create a dictionary which maps the names of the files to functions, seelocales/en.pyfor an example. That can be passed ashandlerstoTakeoutParserTo cache and merge takeouts (maintains a single dependency on the paths you pass -- so if you change the input paths, it does a full recompute)fromgoogle_takeout.mergeimportcached_merge_takeoutsresults=list(cached_merge_takeouts(["/full/path/to/Takeout-1599315526","/full/path/to/Takeout-1634971143"]))If you don't want to cache the results but want to merge results from multiple takeouts, can do something custom by directly using themerge_eventsfunction:fromgoogle_takeout_parser.mergeimportmerge_events,TakeoutParseritrs=[]# list of iterators of google eventsforpathin['path/to/Takeout-1599315526''path/to/Takeout-1616796262']:# ignore errors, error_policy can be 'yield', 'raise' or 'drop'tk=TakeoutParser(path,error_policy="drop")itrs.append(tk.parse(cache=False))res=list(merge_events(*itrs))The events this returns is a combination of all types in themodels.py, to filter to a particular type you can provide that to skip parsing other files:fromgoogle_takeout_parser.modelsimportLocationfromgoogle_takeout_parser.path_dispatchimportTakeoutParser# filter_type can be a list to filter multiple typeslocations=list(TakeoutParser("path/to/Takeout").parse(filter_type=Location))len(locations)99913I personally exclusively use this through theHPI google takeoutfile, as a configuration layer to locate where my takeouts are on disk, and since that 'automatically' unzips the takeouts (I store them as the zips), i.e., doesn't require me to maintain an unpacked viewLegacy HTML ParsingI wouldheavily recommend againstusing the HTML format forMy Activity. It is not always possible to properly parse the metadata, is more prone to errors parsing dates due to local timezones, and takes much longer to parse than the JSON format.On certain machines, the giant HTML files may even take so much memory that the process is eventually killed for using too much memory. For a workaround, seesplit_html.ContributingJust to give a brief overview, to add new functionality (parsing some new folder that this doesn't currently support), you'd need to:Add amodelfor it inmodels.pysubclassingBaseEventand adding it to the Union at the bottom of the file. That should have akeyproperty function which describes each event uniquely (used to merge takeout events)Write a function which takes thePathto the file you're trying to parse and converts it to the model you created (See examples inparse_json.py). Ideally extract a single raw item from the takeout file add a test for it so its obvious when/if the format changes.Add a regex match for the file path to the handler map ingoogle_takeout_parser/locales/en.py.Dont feel required to add support for all locales, its somewhat annoying to swap languages on google, request a takeout, wait for it to process and then swap back.Though, if your takeout is in some language this doesn't support, you cancreate an issuewith the file structure (runfind Takeoutand/ortree Takeout), or contribute a locale file by creating apath -> function mapping, and adding it to the globalLOCALESvariables inlocales/all.pyandlocales/main.pyThis is a pretty difficult to maintain, as it requires a lot of manual testing from people who have access to these takeouts, and who actively use the language that the takeout is in. My google accounts main language is English, so I upkeep that locale whenever I notice changes, but its not trivial to port those changes to other locales without swapping my language, making an export, waiting, and then switching back. I keep track of mismatched changesin this boardIdeally, you would select everything when doing a takeout (not just theMy Activity/Chrome/Location Historylike I suggested above), sopaths that are not parsed can be ignored properly.Testinggitclone'https://github.com/seanbreckenridge/google_takeout_parser'cd./google_takeout_parser
pipinstall'.[testing]'mypy./google_takeout_parser
flake8./google_takeout_parser
pytest