Package: cargo-crates
Cargo CratesAn easy way to build data extractors in Docker.Get it ... it's data containers ... cargo crates.Anyway.UsageConvention over configuration(?).Cargo crates share a common set of patterns that make them easy to use.Each crate is associated with a specific API.Authentication is provided via environment variables.Parameters are provided to the container at runtime.For example, to extract sleep data from the Oura API you would run a container like this:dockerrun-eOURA_PAT=<TOKEN>ghcr.io/dacort/crates-ouraYou can be more specific or select different APIs, too:# Returns data beginning 2021-01-01dockerrun-eOURA_PAT=<TOKEN>-estart=2021-01-01ghcr.io/dacort/crates-oura# Returns activity data instead of sleep data (the default)dockerrun-eOURA_PAT=<TOKEN>ghcr.io/dacort/crates-ouraactivityOther APIs might need more flexibility - for example, with Twitter there are numerous endpoints.In this case, the.envfile contains the environment variables needed for Twitter authentication.# Extract current user infodockerrun--env-file.envghcr.io/dacort/crates-twitterusers/showdacort# Or fetch a user's followersdockerrun--env-file.envghcr.io/dacort/crates-twitterfollowersdacortThe idea is that the Docker container provides an abstraction on top of the underlying APIs and can be implemented however you want.Perhaps someday down the road there could be a set of base images that provide a nice set of abstractions for generic authentication and error handling. But for now, it's just a bunch of python.CaveatsRight now I'm building this for my own purposes. So the set of supported APIs or endpoints is rather limited.Supported ServicesGitHubSupported Commands:traffic- returns results from GitHub traffic stats including clones, popular/paths, popular/referrers, and views. Each type of traffic stat is a subcommand.releases- returns information about releases for a specific GitHub repo.Environment Variables:GITHUB_PAT- Your GitHubPersonal Access TokenExamples:Return all traffic stats fordacort/cargo-cratesdockerrun-eGITHUB_PAT\ghcr.io/dacort/crates-github\trafficdacort/cargo-crates{"repo":"dacort/cargo-crates","path":"clones","stats":{"count":77,"uniques":8,"clones":[{"timestamp":"2021-03-18T00:00:00Z","count":77,"uniques":8}]}}{"repo":"dacort/cargo-crates","path":"popular/paths","stats":[{"path":"/dacort/cargo-crates/actions","title":"Actions \u00b7 dacort/cargo-crates","count":33,"uniques":1},{"path":"/dacort/cargo-crates","title":"dacort/cargo-crates","count":11,"uniques":2},{"path":"/dacort/cargo-crates/actions/workflows/crates.yaml","title":"Actions \u00b7 dacort/cargo-crates","count":7,"uniques":1},{"path":"/dacort/cargo-crates/actions/runs/666130915","title":"Remove useless job0 \u00b7 dacort/cargo-crates@6b54337","count":4,"uniques":1},{"path":"/dacort/cargo-crates/actions/runs/666151009","title":"Trying something else \u00b7 dacort/cargo-crates@ed3d226","count":3,"uniques":1},{"path":"/dacort/cargo-crates/actions/runs/666165537","title":"Hmm \u00b7 dacort/cargo-crates@64c59e7","count":3,"uniques":1},{"path":"/dacort/cargo-crates/actions/runs/666215936","title":"Add requirements \u00b7 dacort/cargo-crates@6161093","count":3,"uniques":1},{"path":"/dacort/cargo-crates/actions/runs/666227882","title":"Does this actually work now?? \u00b7 dacort/cargo-crates@37d31ea","count":3,"uniques":1},{"path":"/dacort/cargo-crates/pulls","title":"Pull requests \u00b7 dacort/cargo-crates","count":3,"uniques":1},{"path":"/dacort/cargo-crates/actions/workflows/test_matrix.yaml","title":"Actions \u00b7 dacort/cargo-crates","count":2,"uniques":1}]}{"repo":"dacort/cargo-crates","path":"popular/referrers","stats":[{"referrer":"github.com","count":3,"uniques":2}]}{"repo":"dacort/cargo-crates","path":"views","stats":{"count":108,"uniques":2,"views":[{"timestamp":"2021-03-18T00:00:00Z","count":108,"uniques":2}]}}Return only views fordacort/cargo-cratesdockerrun-eGITHUB_PATghcr.io/dacort/crates-githubtrafficdacort/cargo-cratesviews{"repo":"dacort/cargo-crates","path":"views","stats":{"count":108,"uniques":2,"views":[{"timestamp":"2021-03-18T00:00:00Z","count":108,"uniques":2}]}}YouTubeSupported Commands:videos- return video stats for up to 50 video IDs.channel_videos- return video stats for 50 videos on the provided channel ID.Environment Variables:YOUTUBE_API_KEY- An API key for theYouTube Data APIExamples:Return video stats forIntro to EMR Studiodockerrun-eYOUTUBE_API_KEY\ghcr.io/dacort/crates-youtube\videosrZ3zeJ6WKPYClick to view JSON```json
  {
      "kind": "youtube#video",
      "etag": "tbhtHqbvEMe3jNGfFGK36HpowJk",
      "id": "rZ3zeJ6WKPY",
      "snippet": {
          "publishedAt": "2021-04-21T22:43:26Z",
          "channelId": "UCd6MoB9NC6uYN2grvUNT-Zg",
          "title": "Intro to Amazon EMR Studio",
          "description": "Introduction to Amazon EMR Studio. In this video, we show how to:\n- Create a new workspace\n- Utilize cluster templates for EMR clusters\n- Connect EMR Studio to a GitHub repository\n- Execute parameterized notebooks\n\nLearn more about Amazon EMR Studio - https://amzn.to/3xhbuEj\n\nSubscribe: \nMore AWS videos - http://bit.ly/2O3zS75 \nMore AWS events videos - http://bit.ly/316g9t4\n\n#AWS #AWSDemo #AmazonEMRStudio",
          "thumbnails": {
              "default": {
                  "url": "https://i.ytimg.com/vi/rZ3zeJ6WKPY/default.jpg",
                  "width": 120,
                  "height": 90
              },
              "medium": {
                  "url": "https://i.ytimg.com/vi/rZ3zeJ6WKPY/mqdefault.jpg",
                  "width": 320,
                  "height": 180
              },
              "high": {
                  "url": "https://i.ytimg.com/vi/rZ3zeJ6WKPY/hqdefault.jpg",
                  "width": 480,
                  "height": 360
              },
              "standard": {
                  "url": "https://i.ytimg.com/vi/rZ3zeJ6WKPY/sddefault.jpg",
                  "width": 640,
                  "height": 480
              },
              "maxres": {
                  "url": "https://i.ytimg.com/vi/rZ3zeJ6WKPY/maxresdefault.jpg",
                  "width": 1280,
                  "height": 720
              }
          },
          "channelTitle": "Amazon Web Services",
          "tags": [
              "AWS",
              "Amazon Web Services",
              "Cloud",
              "AWS Cloud",
              "Cloud Computing",
              "Amazon AWS",
              "Amazon EMR",
              "Amazon EMR Studio",
              "Jupyter"
          ],
          "categoryId": "28",
          "liveBroadcastContent": "none",
          "localized": {
              "title": "Intro to Amazon EMR Studio",
              "description": "Introduction to Amazon EMR Studio. In this video, we show how to:\n- Create a new workspace\n- Utilize cluster templates for EMR clusters\n- Connect EMR Studio to a GitHub repository\n- Execute parameterized notebooks\n\nLearn more about Amazon EMR Studio - https://amzn.to/3xhbuEj\n\nSubscribe: \nMore AWS videos - http://bit.ly/2O3zS75 \nMore AWS events videos - http://bit.ly/316g9t4\n\n#AWS #AWSDemo #AmazonEMRStudio"
          },
          "defaultAudioLanguage": "en"
      },
      "contentDetails": {
          "duration": "PT12M34S",
          "dimension": "2d",
          "definition": "hd",
          "caption": "false",
          "licensedContent": false,
          "contentRating": {},
          "projection": "rectangular"
      },
      "statistics": {
          "viewCount": "537",
          "likeCount": "9",
          "dislikeCount": "0",
          "favoriteCount": "0",
          "commentCount": "0"
      }
  }
  ```OuraRetrieve one of three differentdaily summariesfrom the Oura Ring API.Supported Commands:sleep- returnsleep periodsactivity- returnactivity summaryreadiness- returnreadiness dataEnvironment Variables:OURA_PAT- Your Ourapersonal access tokenExamples:Return sleep data for the past 7 daysdockerrun-eOURA_PAT\ghcr.io/dacort/crates-oura\sleep|head-n1Click to view JSON```json
  {
      "awake": 2010,
      "bedtime_end": "2021-04-20T06:55:43-07:00",
      "bedtime_end_delta": 24943,
      "bedtime_start": "2021-04-19T23:24:43-07:00",
      "bedtime_start_delta": -2117,
      "breath_average": 16.25,
      "deep": 4290,
      "duration": 27060,
      "efficiency": 93,
      "hr_5min": [
          63,
          63,
          64,
          65,
          66,
          66,
          66,
          67,
          66,
          66,
          67,
          70,
          71,
          70,
          69,
          69,
          70,
          69,
          68,
          68,
          68,
          67,
          67,
          66,
          66,
          66,
          66,
          66,
          62,
          64,
          70,
          68,
          67,
          66,
          66,
          67,
          66,
          66,
          65,
          66,
          66,
          67,
          67,
          67,
          67,
          70,
          69,
          71,
          71,
          69,
          68,
          66,
          63,
          65,
          67,
          66,
          65,
          64,
          63,
          62,
          63,
          65,
          66,
          65,
          69,
          72,
          74,
          72,
          68,
          68,
          69,
          68,
          66,
          62,
          63,
          62,
          61,
          60,
          60,
          58,
          60,
          57,
          58,
          57,
          57,
          58,
          61,
          64,
          66,
          61,
          0
      ],
      "hr_average": 65.68,
      "hr_lowest": 57,
      "hypnogram_5min": "4221111111222322221111222122223332222222223344333332222212222112333334233332222242222222444",
      "is_longest": 1,
      "light": 14730,
      "midpoint_at_delta": 11083,
      "midpoint_time": 13200,
      "onset_latency": 270,
      "period_id": 0,
      "rem": 6030,
      "restless": 29,
      "rmssd": 27,
      "rmssd_5min": [
          33,
          27,
          20,
          19,
          17,
          17,
          17,
          17,
          19,
          22,
          31,
          28,
          18,
          19,
          20,
          19,
          14,
          15,
          17,
          14,
          14,
          15,
          15,
          18,
          17,
          17,
          16,
          40,
          44,
          32,
          21,
          26,
          30,
          27,
          20,
          20,
          17,
          27,
          19,
          16,
          18,
          22,
          23,
          24,
          33,
          23,
          27,
          19,
          19,
          19,
          36,
          29,
          38,
          29,
          27,
          27,
          36,
          31,
          29,
          34,
          29,
          28,
          24,
          36,
          26,
          23,
          19,
          17,
          31,
          30,
          29,
          30,
          32,
          47,
          39,
          30,
          44,
          53,
          48,
          41,
          42,
          65,
          47,
          62,
          48,
          39,
          36,
          22,
          26,
          36,
          0
      ],
      "score": 83,
      "score_alignment": 97,
      "score_deep": 88,
      "score_disturbances": 77,
      "score_efficiency": 98,
      "score_latency": 72,
      "score_rem": 91,
      "score_total": 77,
      "summary_date": "2021-04-19",
      "temperature_delta": 0.06,
      "temperature_deviation": 0.06,
      "temperature_trend_deviation": 0.29,
      "timezone": -420,
      "total": 25050,
      "activity_type": "sleep"
  }
  ```Reddit APII wanted to get my saved posts out of theReddit API. It's a little tough to understand, and the saved posts aren't actually documented there...but aftersome hunting, I found that they can be accessed at/user/dacort/saved.json.This functionality just fetches the most recent 100 saved posts every time. I considered adding astart_datefilter, but don't need it right now. 😁Supported Commands:saved  - list saved posts for<username>search [subreddit] <search_term> - search all of Reddit or a specific subreddit (r/subreddit_name) for a search termNote that terms in quotes need to be surrounded in single quotes, e.g.'"cargo crates"'Environment Variables:CLIENT_ID - Client ID of a"script"type appCLIENT_SECRET - App secretUSERNAME - Your Reddit usernamePASSWORD - Your Reddit passwordSlack Web APITheSlack Web APIis intended for use with ad-hoc queries and I use it to query basic info about some of the channels I'm in.Supported Commands:channels-list all channelsin a Slack teamsearch-search for a keywordin your Slack teamEnvironment Variables:SLACK_TOKEN- Some sort of Slack token that starts withxox...Examples:Get a list of channels for the workspace the token is associated withdockerrun-eSLACK_TOKEN\ghcr.io/dacort/crates-slackchannelsPerform a search in the workspace the token is associated withdockerrun-eSLACK_TOKEN\ghcr.io/dacort/crates-slacksearchdacortTwitterOnly a couple Twitter endpoints are defined at this point.Supported Commands:followers- return list of followers for a given screen nameusers/show- return user profile data for a given screen nameEnvironment Variables - create a Twitter app and define the 4 different variables.CONSUMER_KEYCONSUMER_SECRETACCESS_TOKEN_KEYACCESS_TOKEN_SECRETYou can store all these environment variables in a file and reference that file with Docker.Examples:Return@dacort'sprofiledockerrun--env-file.env\ghcr.io/dacort/crates-twitter\users/showdacort{"id":"99723","name":"Damon Cortesi","username":"dacort"}Writing output to Amazon S3The original idea behind this repo was that I could run these containers and easily output the resulting data to templated paths on S3.So I built another tool, calledForklift, that comes bundled with each Cargo Crate.If you run the container normally, the output gets printed to stdout. However, if you provide aFORKLIFT_URIenvironment variable,
the data will get written to the S3 path provided. You can templatize parts of the path with JSON keys or a couple helper functions.For example, the command below will upload my profile info tos3://<BUCKET>/forklift/twitter/dt=2021-05-15/dacort.jsondockerrun\-eFORKLIFT_URI='s3://<BUCKET>/forklift/twitter/dt={{ today }}/{{json "username"}}.json'\-eAWS_ACCESS_KEY_ID\-eAWS_SECRET_ACCESS_KEY\--env-file.env\ghcr.io/dacort/crates-twitterusers/showdacortNote that we passed AWS credentials as environment variables. You will either need to do that or run the container in AWS usinginstance profiles.Some higher level thoughts after a few implementationsThe data is intended to be streamedOutput is always a JSON streamEach implementation is intended to be completely independentThey're all Python...they could be anything else!Each implementation is intended to be as lightweight as possibleDon't force unnecessary requirements into a crateNext stepsFormalize the executor API and base classI want a way to be able to register commands, their parameters, and required environment variablesFormalize the output API and stdout/S3 classesRight now it's pretty hardcoded into the GitHub crateMaybe somedayA generic "auth" command for cratesResourcesPython project structureMinimal Docker containers for Python(ended up not using, but)IO Streams with PythonFor building matrix builds