Repository: ibis-project/ibis

Issue:
refactor: remove ibis.NA 
Opening an issue to discuss a separate the refactor of `ibis.NA` because doing it as part of https://github.com/ibis-project/ibis/pull/9300 will get messy. 

To discuss:
Does removing `ibis.NA`imply:

1. Replace `ibis.NA` for `ibis.null()`
2. Get rid of untyped nulls all together, as in not use `ibis.null()` but just None

Discussion:
github-actions[bot]: **ACTION NEEDED**

Ibis follows the [Conventional Commits specification](https://www.conventionalcommits.org/en/v1.0.0/) for release automation.

The PR title and description are used as the merge commit message.

Please update your PR title and description to match the specification.

Patch:
diff --git a/.github/workflows/ibis-docs-main.yml b/.github/workflows/ibis-docs-main.yml
index 95317ba1c..49eb77ed8 100644
--- a/.github/workflows/ibis-docs-main.yml
+++ b/.github/workflows/ibis-docs-main.yml
@@ -74,5 +74,5 @@ jobs:
           python .github/workflows/upload-algolia.py
         env:
           ALGOLIA_WRITE_API_KEY: ${{ secrets.ALGOLIA_WRITE_API_KEY }}
-          ALGOLIA_APP_ID: HS77W8GWM1
+          ALGOLIA_APP_ID: TNU9HG3L41
           ALGOLIA_INDEX: prod_ibis
diff --git a/docs/_freeze/posts/campaign-finance/index/execute-results/html.json b/docs/_freeze/posts/campaign-finance/index/execute-results/html.json
index 4da314f90..4f436cb38 100644
--- a/docs/_freeze/posts/campaign-finance/index/execute-results/html.json
+++ b/docs/_freeze/posts/campaign-finance/index/execute-results/html.json
@@ -1,14 +1,15 @@
 {
-  "hash": "2631514785c59e4e1d3b37b9c07ea232",
+  "hash": "989ed0f2ebddb8e202db6a33bc1bf790",
   "result": {
-    "markdown": "---\ntitle: \"Exploring campaign finance data\"\nauthor: \"Nick Crews\"\ndate: \"2023-03-24\"\ncategories:\n    - blog\n    - data engineering\n    - case study\n    - duckdb\n    - performance\n---\n\nHi! My name is [Nick Crews](https://www.linkedin.com/in/nicholas-b-crews/),\nand I'm a data engineer that looks at public campaign finance data.\n\nIn this post, I'll walk through how I use Ibis to explore public campaign contribution\ndata from the Federal Election Commission (FEC). We'll do some loading,\ncleaning, featurizing, and visualization. There will be filtering, sorting, grouping,\nand aggregation.\n\n## Downloading The Data\n\n::: {#02d63441 .cell execution_count=1}\n``` {.python .cell-code}\nfrom pathlib import Path\nfrom zipfile import ZipFile\nfrom urllib.request import urlretrieve\n\n# Download and unzip the 2018 individual contributions data\nurl = \"https://cg-519a459a-0ea3-42c2-b7bc-fa1143481f74.s3-us-gov-west-1.amazonaws.com/bulk-downloads/2018/indiv18.zip\"\nzip_path = Path(\"indiv18.zip\")\ncsv_path = Path(\"indiv18.csv\")\n\nif not zip_path.exists():\n    urlretrieve(url, zip_path)\n\nif not csv_path.exists():\n    with ZipFile(zip_path) as zip_file, csv_path.open(\"w\") as csv_file:\n        for line in zip_file.open(\"itcont.txt\"):\n            csv_file.write(line.decode())\n```\n:::\n\n\n## Loading the data\n\nNow that we have our raw data in a .csv format, let's load it into Ibis,\nusing the duckdb backend.\n\nNote that a 4.3 GB .csv would be near the limit of what pandas could\nhandle on my laptop with 16GB of RAM. In pandas, typically every time\nyou perform a transformation on the data, a copy of the data is made.\nI could only do a few transformations before I ran out of memory.\n\nWith Ibis, this problem is solved in two different ways.\n\nFirst, because they are designed to work with very large datasets,\nmany (all?) SQL backends support out of core operations.\nThe data lives on disk, and are only loaded in a streaming fashion\nwhen needed, and then written back to disk as the operation is performed.\n\nSecond, unless you explicitly ask for it, Ibis makes use of lazy\nevaluation. This means that when you ask for a result, the\nresult is not persisted in memory. Only the original source\ndata is persisted. Everything else is derived from this on the fly.\n\n::: {#83a871f2 .cell execution_count=2}\n``` {.python .cell-code}\nimport ibis\nfrom ibis import _\n\nibis.options.interactive = True\n\n# The raw .csv file doesn't have column names, so we will add them in the next step.\nraw = ibis.read_csv(csv_path)\nraw\n```\n\n::: {.cell-output .cell-output-display execution_count=2}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> C00401224 </span>┃<span style=\"font-weight: bold\"> A      </span>┃<span style=\"font-weight: bold\"> M6     </span>┃<span style=\"font-weight: bold\"> P      </span>┃<span style=\"font-weight: bold\"> 201804059101866001 </span>┃<span style=\"font-weight: bold\"> 24T    </span>┃<span style=\"font-weight: bold\"> IND    </span>┃<span style=\"font-weight: bold\"> STOUFFER, LEIGH   </span>┃<span style=\"font-weight: bold\"> AMSTELVEEN   </span>┃<span style=\"font-weight: bold\"> ZZ     </span>┃<span style=\"font-weight: bold\"> 1187RC    </span>┃<span style=\"font-weight: bold\"> MYSELF            </span>┃<span style=\"font-weight: bold\"> SELF EMPLOYED           </span>┃<span style=\"font-weight: bold\"> 05172017 </span>┃<span style=\"font-weight: bold\"> 10    </span>┃<span style=\"font-weight: bold\"> C00458000 </span>┃<span style=\"font-weight: bold\"> SA11AI_81445687 </span>┃<span style=\"font-weight: bold\"> 1217152 </span>┃<span style=\"font-weight: bold\"> column18 </span>┃<span style=\"font-weight: bold\"> EARMARKED FOR PROGRESSIVE CHANGE CAMPAIGN COMMITTEE (C00458000) </span>┃<span style=\"font-weight: bold\"> 4050820181544765358 </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>              │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                  │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>   │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>   │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>   │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                                          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>               │\n├───────────┼────────┼────────┼────────┼────────────────────┼────────┼────────┼───────────────────┼──────────────┼────────┼───────────┼───────────────────┼─────────────────────────┼──────────┼───────┼───────────┼─────────────────┼─────────┼──────────┼─────────────────────────────────────────────────────────────────┼─────────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101867748</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STRAWS, JOYCE    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">34761    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SILVERSEA CRUISES</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">RESERVATIONS SUPERVISOR</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05182017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81592336</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544770597</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101867748</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STRAWS, JOYCE    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">34761    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SILVERSEA CRUISES</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">RESERVATIONS SUPERVISOR</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81627562</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544770598</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865942</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">039020760</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE             </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE                   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05132017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81047921</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765179</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865942</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">039020760</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE             </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE                   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05152017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81209209</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765180</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865942</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">039020760</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE             </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE                   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81605223</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765181</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865943</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">039020760</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE             </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE                   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05242017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_82200022</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765182</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865943</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">03902    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED           </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05292017</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00213512</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_82589834</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR NANCY PELOSI FOR CONGRESS (C00213512)            </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765184</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865944</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">039020760</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE             </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE                   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05302017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_82643727</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765185</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101867050</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STRANGE, WINIFRED</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">34216    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED           </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05162017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81325918</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544768505</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101867051</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STRANGE, WINIFRED</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">34216    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED           </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05232017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81991189</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544768506</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                 │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                 │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>        │     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                                               │                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└───────────┴────────┴────────┴────────┴────────────────────┴────────┴────────┴───────────────────┴──────────────┴────────┴───────────┴───────────────────┴─────────────────────────┴──────────┴───────┴───────────┴─────────────────┴─────────┴──────────┴─────────────────────────────────────────────────────────────────┴─────────────────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#d2a81789 .cell execution_count=3}\n``` {.python .cell-code}\n# For a more comprehesive description of the columns and their meaning, see\n# https://www.fec.gov/campaign-finance-data/contributions-individuals-file-description/\ncolumns = {\n    \"CMTE_ID\": \"keep\",  # Committee ID\n    \"AMNDT_IND\": \"drop\",  # Amendment indicator. A = amendment, N = new, T = termination\n    \"RPT_TP\": \"drop\",  # Report type (monthly, quarterly, etc)\n    \"TRANSACTION_PGI\": \"keep\",  # Primary/general indicator\n    \"IMAGE_NUM\": \"drop\",  # Image number\n    \"TRANSACTION_TP\": \"drop\",  # Transaction type\n    \"ENTITY_TP\": \"keep\",  # Entity type\n    \"NAME\": \"drop\",  # Contributor name\n    \"CITY\": \"keep\",  # Contributor city\n    \"STATE\": \"keep\",  # Contributor state\n    \"ZIP_CODE\": \"drop\",  # Contributor zip code\n    \"EMPLOYER\": \"drop\",  # Contributor employer\n    \"OCCUPATION\": \"drop\",  # Contributor occupation\n    \"TRANSACTION_DT\": \"keep\",  # Transaction date\n    \"TRANSACTION_AMT\": \"keep\",  # Transaction amount\n    # Other ID. For individual contributions will be null. For contributions from\n    # other FEC committees, will be the committee ID of the other committee.\n    \"OTHER_ID\": \"drop\",\n    \"TRAN_ID\": \"drop\",  # Transaction ID\n    \"FILE_NUM\": \"drop\",  # File number, unique number assigned to each report filed with the FEC\n    \"MEMO_CD\": \"drop\",  # Memo code\n    \"MEMO_TEXT\": \"drop\",  # Memo text\n    \"SUB_ID\": \"drop\",  # Submission ID. Unique number assigned to each transaction.\n}\n\nrenaming = {old: new for old, new in zip(raw.columns, columns.keys())}\nto_keep = [k for k, v in columns.items() if v == \"keep\"]\nkept = raw.relabel(renaming)[to_keep]\nkept\n```\n\n::: {.cell-output .cell-output-display execution_count=3}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CMTE_ID   </span>┃<span style=\"font-weight: bold\"> TRANSACTION_PGI </span>┃<span style=\"font-weight: bold\"> ENTITY_TP </span>┃<span style=\"font-weight: bold\"> CITY         </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_DT </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │\n├───────────┼─────────────────┼───────────┼──────────────┼────────┼────────────────┼─────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05182017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05132017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05152017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017      </span> │               <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05242017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05292017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05302017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05162017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05232017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>              │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└───────────┴─────────────────┴───────────┴──────────────┴────────┴────────────────┴─────────────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#1e6d16fe .cell execution_count=4}\n``` {.python .cell-code}\n# 21 million rows\nkept.count()\n```\n\n::: {.cell-output .cell-output-display}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=4}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-cyan-fg ansi-bold\">21730730</span></pre>\n```\n:::\n\n:::\n:::\n\n\nHuh, what's up with those timings? Previewing the head only took a fraction of a second,\nbut finding the number of rows took 10 seconds.\n\nThat's because duckdb is scanning the .csv file on the fly every time we access it.\nSo we only have to read the first few lines to get that preview,\nbut we have to read the whole file to get the number of rows.\n\nNote that this isn't a feature of Ibis, but a feature of Duckdb. This what I think is\none of the strengths of Ibis: Ibis itself doesn't have to implement any of the\noptimimizations or features of the backends. Those backends can focus on what they do\nbest, and Ibis can get those things for free.\n\nSo, let's tell duckdb to actually read in the file to its native format so later accesses\nwill be faster. This will be a ~20 seconds that we'll only have to pay once.\n\n::: {#185a2d89 .cell execution_count=5}\n``` {.python .cell-code}\nkept = kept.cache()\nkept\n```\n\n::: {.cell-output .cell-output-display execution_count=5}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CMTE_ID   </span>┃<span style=\"font-weight: bold\"> TRANSACTION_PGI </span>┃<span style=\"font-weight: bold\"> ENTITY_TP </span>┃<span style=\"font-weight: bold\"> CITY         </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_DT </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │\n├───────────┼─────────────────┼───────────┼──────────────┼────────┼────────────────┼─────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05182017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05132017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05152017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017      </span> │               <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05242017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05292017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05302017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05162017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05232017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>              │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└───────────┴─────────────────┴───────────┴──────────────┴────────┴────────────────┴─────────────────┘\n</pre>\n```\n:::\n:::\n\n\nLook, now accessing it only takes a fraction of a second!\n\n::: {#9253e73f .cell execution_count=6}\n``` {.python .cell-code}\nkept.count()\n```\n\n::: {.cell-output .cell-output-display}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=6}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-cyan-fg ansi-bold\">21730730</span></pre>\n```\n:::\n\n:::\n:::\n\n\n### Committees Data\n\nThe contributions only list an opaque `CMTE_ID` column. We want to know which actual\ncommittee this is. Let's load the committees table so we can lookup from\ncommittee ID to committee name.\n\n::: {#30076e2c .cell execution_count=7}\n``` {.python .cell-code}\ndef read_committees():\n    committees_url = \"https://cg-519a459a-0ea3-42c2-b7bc-fa1143481f74.s3-us-gov-west-1.amazonaws.com/bulk-downloads/2018/committee_summary_2018.csv\"\n    # This just creates a view, it doesn't actually fetch the data yet\n    tmp = ibis.read_csv(committees_url)\n    tmp = tmp[\"CMTE_ID\", \"CMTE_NM\"]\n    # The raw table contains multiple rows for each committee id, so lets pick\n    # an arbitrary row for each committee id as the representative name.\n    deduped = tmp.group_by(\"CMTE_ID\").agg(CMTE_NM=_.CMTE_NM.arbitrary())\n    return deduped\n\n\ncomms = read_committees().cache()\ncomms\n```\n\n::: {.cell-output .cell-output-display execution_count=7}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CMTE_ID   </span>┃<span style=\"font-weight: bold\"> CMTE_NM                                                        </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                                         │\n├───────────┼────────────────────────────────────────────────────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00659441</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">JASON ORTITAY FOR CONGRESS                                    </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00661249</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SERVICE AFTER SERVICE                                         </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00457754</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">U.S. TRAVEL ASSOCIATION PAC                                   </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00577635</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ISAKSON VICTORY COMMITTEE                                     </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00297911</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">TEXAS FORESTRY ASSOCIATION FORESTRY POLITICAL ACTION COMMITTEE</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00551382</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">VOTECLIMATE.US PAC                                            </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00414318</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">LOEBSACK FOR CONGRESS                                         </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00610709</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">AUSTIN INNOVATION 2016                                        </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00131607</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FLORIDA CITRUS MUTUAL POLITCAL ACTION COMMITTEE               </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00136531</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NATIONAL DEMOCRATIC POLICY COMMITTEE                          </span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                                              │\n└───────────┴────────────────────────────────────────────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\nNow add the committee name to the contributions table:\n\n::: {#0a9f3b35 .cell execution_count=8}\n``` {.python .cell-code}\ntogether = kept.left_join(comms, \"CMTE_ID\").drop(\"CMTE_ID\", \"CMTE_ID_right\")\ntogether\n```\n\n::: {.cell-output .cell-output-display execution_count=8}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> TRANSACTION_PGI </span>┃<span style=\"font-weight: bold\"> ENTITY_TP </span>┃<span style=\"font-weight: bold\"> CITY             </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_DT </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃<span style=\"font-weight: bold\"> CMTE_NM                                         </span>┃\n┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                          │\n├─────────────────┼───────────┼──────────────────┼────────┼────────────────┼─────────────────┼─────────────────────────────────────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">COHASSET        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">MA    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">KEY LARGO       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01042017      </span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5000</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">LOOKOUT MOUNTAIN</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NORTH YARMOUTH  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ALPHARETTA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">HOLLIS CENTER   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ALEXANDRIA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">VA    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>              │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                               │\n└─────────────────┴───────────┴──────────────────┴────────┴────────────────┴─────────────────┴─────────────────────────────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\n## Cleaning\n\nFirst, let's drop any contributions that don't have a committee name. There are only 6 of them.\n\n::: {#14ae871f .cell execution_count=9}\n``` {.python .cell-code}\n# We can do this fearlessly, no .copy() needed, because\n# everything in Ibis is immutable. If we did this in pandas,\n# we might start modifying the original DataFrame accidentally!\ncleaned = together\n\nhas_name = cleaned.CMTE_NM.notnull()\ncleaned = cleaned[has_name]\nhas_name.value_counts()\n```\n\n::: {.cell-output .cell-output-display execution_count=9}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> NotNull(CMTE_NM) </span>┃<span style=\"font-weight: bold\"> NotNull(CMTE_NM)_count </span>┃\n┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">boolean</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>                  │\n├──────────────────┼────────────────────────┤\n│ True             │               <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21730724</span> │\n│ False            │                      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span> │\n└──────────────────┴────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\nLet's look at the `ENTITY_TP` column. This represents the type of entity that\nmade the contribution:\n\n::: {#72577ed8 .cell execution_count=10}\n``` {.python .cell-code}\ntogether.ENTITY_TP.value_counts()\n```\n\n::: {.cell-output .cell-output-display execution_count=10}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> ENTITY_TP </span>┃<span style=\"font-weight: bold\"> ENTITY_TP_count </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │\n├───────────┼─────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21687992</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">CCM      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">698</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">CAN      </span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13659</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">ORG      </span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18555</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">PTY      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">49</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">COM      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">867</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">PAC      </span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3621</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>      │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5289</span> │\n└───────────┴─────────────────┘\n</pre>\n```\n:::\n:::\n\n\nWe only care about contributions from individuals.\n\nOnce we filter on this column, the contents of it are irrelevant, so let's drop it.\n\n::: {#f29924a2 .cell execution_count=11}\n``` {.python .cell-code}\ncleaned = together[_.ENTITY_TP == \"IND\"].drop(\"ENTITY_TP\")\n```\n:::\n\n\nIt looks like the `TRANSACTION_DT` column was a raw string like \"MMDDYYYY\",\nso let's convert that to a proper date type.\n\n::: {#15443483 .cell execution_count=12}\n``` {.python .cell-code}\nfrom ibis.expr.types import StringValue, DateValue\n\n\ndef mmddyyyy_to_date(val: StringValue) -> DateValue:\n    return val.cast(str).lpad(8, \"0\").to_timestamp(\"%m%d%Y\").date()\n\n\ncleaned = cleaned.mutate(date=mmddyyyy_to_date(_.TRANSACTION_DT)).drop(\"TRANSACTION_DT\")\ncleaned\n```\n\n::: {.cell-output .cell-output-display execution_count=12}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> TRANSACTION_PGI </span>┃<span style=\"font-weight: bold\"> CITY             </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃<span style=\"font-weight: bold\"> CMTE_NM                                         </span>┃<span style=\"font-weight: bold\"> date       </span>┃\n┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">date</span>       │\n├─────────────────┼──────────────────┼────────┼─────────────────┼─────────────────────────────────────────────────┼────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">COHASSET        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">MA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">KEY LARGO       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5000</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-04</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">LOOKOUT MOUNTAIN</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NORTH YARMOUTH  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ALPHARETTA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">HOLLIS CENTER   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ALEXANDRIA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">VA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>          │\n└─────────────────┴──────────────────┴────────┴─────────────────┴─────────────────────────────────────────────────┴────────────┘\n</pre>\n```\n:::\n:::\n\n\nThe `TRANSACTION_PGI` column represents the type (primary, general, etc) of election,\nand the year. But it seems to be not very consistent:\n\n::: {#fa016097 .cell execution_count=13}\n``` {.python .cell-code}\ncleaned.TRANSACTION_PGI.topk(10)\n```\n\n::: {.cell-output .cell-output-display execution_count=13}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> TRANSACTION_PGI </span>┃<span style=\"font-weight: bold\"> Count(TRANSACTION_PGI) </span>┃\n┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>                  │\n├─────────────────┼────────────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │               <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">17013596</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">G2018          </span> │                <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2095123</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P2018          </span> │                <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1677183</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P2020          </span> │                 <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">208501</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">O2018          </span> │                 <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">161874</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">S2017          </span> │                 <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">124336</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">G2017          </span> │                  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">98401</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P2022          </span> │                  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">91136</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P2017          </span> │                  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">61153</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">R2017          </span> │                  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">54281</span> │\n└─────────────────┴────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#35c8a393 .cell execution_count=14}\n``` {.python .cell-code}\ndef get_election_type(pgi: StringValue) -> StringValue:\n    \"\"\"Use the first letter of the TRANSACTION_PGI column to determine the election type\n\n    If the first letter is not one of the known election stage, then return null.\n    \"\"\"\n    election_types = {\n        \"P\": \"primary\",\n        \"G\": \"general\",\n        \"O\": \"other\",\n        \"C\": \"convention\",\n        \"R\": \"runoff\",\n        \"S\": \"special\",\n        \"E\": \"recount\",\n    }\n    first_letter = pgi[0]\n    return first_letter.substitute(election_types, else_=ibis.NA)\n\n\ncleaned = cleaned.mutate(election_type=get_election_type(_.TRANSACTION_PGI)).drop(\n    \"TRANSACTION_PGI\"\n)\ncleaned\n```\n\n::: {.cell-output .cell-output-display execution_count=14}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CITY             </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃<span style=\"font-weight: bold\"> CMTE_NM                                         </span>┃<span style=\"font-weight: bold\"> date       </span>┃<span style=\"font-weight: bold\"> election_type </span>┃\n┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">date</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │\n├──────────────────┼────────┼─────────────────┼─────────────────────────────────────────────────┼────────────┼───────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">COHASSET        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">MA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">KEY LARGO       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5000</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-04</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">LOOKOUT MOUNTAIN</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">NORTH YARMOUTH  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">ALPHARETTA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">HOLLIS CENTER   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">ALEXANDRIA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">VA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │\n└──────────────────┴────────┴─────────────────┴─────────────────────────────────────────────────┴────────────┴───────────────┘\n</pre>\n```\n:::\n:::\n\n\nThat worked well! There are 0 nulls in the resulting column, so we always were\nable to determine the election type.\n\n::: {#e7038c36 .cell execution_count=15}\n``` {.python .cell-code}\ncleaned.election_type.topk(10)\n```\n\n::: {.cell-output .cell-output-display execution_count=15}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> election_type </span>┃<span style=\"font-weight: bold\"> Count(election_type) </span>┃\n┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>                │\n├───────────────┼──────────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">19061953</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2216685</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">other        </span> │               <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">161965</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">special      </span> │               <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">149572</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">runoff       </span> │                <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">69637</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">convention   </span> │                <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">22453</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">recount      </span> │                 <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5063</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>          │                    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span> │\n└───────────────┴──────────────────────┘\n</pre>\n```\n:::\n:::\n\n\nAbout 1/20 of transactions are negative. These could represent refunds, or they\ncould be data entry errors. Let's drop them to keep it simple.\n\n::: {#ab64b9b2 .cell execution_count=16}\n``` {.python .cell-code}\nabove_zero = cleaned.TRANSACTION_AMT > 0\ncleaned = cleaned[above_zero]\nabove_zero.value_counts()\n```\n\n::: {.cell-output .cell-output-display execution_count=16}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> Greater(TRANSACTION_AMT, 0) </span>┃<span style=\"font-weight: bold\"> Greater(TRANSACTION_AMT, 0)_count </span>┃\n┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">boolean</span>                     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>                             │\n├─────────────────────────────┼───────────────────────────────────┤\n│ True                        │                          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20669809</span> │\n│ False                       │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1018183</span> │\n└─────────────────────────────┴───────────────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\n## Adding Features\n\nNow that the data is cleaned up to a usable format, let's add some features.\n\nFirst, it's useful to categorize donations by size, placing them into buckets\nof small, medium, large, etc.\n\n::: {#db1e9cbe .cell execution_count=17}\n``` {.python .cell-code}\nedges = [\n    10,\n    50,\n    100,\n    500,\n    1000,\n    5000,\n]\nlabels = [\n    \"<10\",\n    \"10-50\",\n    \"50-100\",\n    \"100-500\",\n    \"500-1000\",\n    \"1000-5000\",\n    \"5000+\",\n]\n\n\ndef bucketize(vals, edges, str_labels):\n    # Uses Ibis's .bucket() method to create a categorical column\n    int_labels = vals.bucket(edges, include_under=True, include_over=True)\n    # Map the integer labels to the string labels\n    int_to_str = {str(i): s for i, s in enumerate(str_labels)}\n    return int_labels.cast(str).substitute(int_to_str)\n\n\nfeatured = cleaned.mutate(amount_bucket=bucketize(_.TRANSACTION_AMT, edges, labels))\nfeatured\n```\n\n::: {.cell-output .cell-output-display execution_count=17}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CITY             </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃<span style=\"font-weight: bold\"> CMTE_NM                                         </span>┃<span style=\"font-weight: bold\"> date       </span>┃<span style=\"font-weight: bold\"> election_type </span>┃<span style=\"font-weight: bold\"> amount_bucket </span>┃\n┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">date</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │\n├──────────────────┼────────┼─────────────────┼─────────────────────────────────────────────────┼────────────┼───────────────┼───────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">COHASSET        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">MA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">KEY LARGO       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5000</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-04</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">1000-5000    </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">LOOKOUT MOUNTAIN</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">NORTH YARMOUTH  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">ALPHARETTA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">HOLLIS CENTER   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">ALEXANDRIA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">VA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │\n└──────────────────┴────────┴─────────────────┴─────────────────────────────────────────────────┴────────────┴───────────────┴───────────────┘\n</pre>\n```\n:::\n:::\n\n\n## Analysis\n\n### By donation size\n\nOne thing we can look at is the donation breakdown by size:\n- Are most donations small or large?\n- Where do politicians/committees get most of their money from? Large or small donations?\n\nWe also will compare performance of Ibis vs pandas during this groupby.\n\n::: {#2c306d0f .cell execution_count=18}\n``` {.python .cell-code}\ndef summary_by(table, by):\n    return table.group_by(by).agg(\n        n_donations=_.count(),\n        total_amount=_.TRANSACTION_AMT.sum(),\n        mean_amount=_.TRANSACTION_AMT.mean(),\n        median_amount=_.TRANSACTION_AMT.approx_median(),\n    )\n\n\ndef summary_by_pandas(df, by):\n    return df.groupby(by, as_index=False).agg(\n        n_donations=(\"election_type\", \"count\"),\n        total_amount=(\"TRANSACTION_AMT\", \"sum\"),\n        mean_amount=(\"TRANSACTION_AMT\", \"mean\"),\n        median_amount=(\"TRANSACTION_AMT\", \"median\"),\n    )\n\n\n# persist the input data so the following timings of the group_by are accurate.\nsubset = featured[\"election_type\", \"amount_bucket\", \"TRANSACTION_AMT\"]\nsubset = subset.cache()\npandas_subset = subset.execute()\n```\n:::\n\n\nLet's take a look at what we are actually computing:\n\n::: {#a621ca5f .cell execution_count=19}\n``` {.python .cell-code}\nby_type_and_bucket = summary_by(subset, [\"election_type\", \"amount_bucket\"])\nby_type_and_bucket\n```\n\n::: {.cell-output .cell-output-display execution_count=19}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> election_type </span>┃<span style=\"font-weight: bold\"> amount_bucket </span>┃<span style=\"font-weight: bold\"> n_donations </span>┃<span style=\"font-weight: bold\"> total_amount </span>┃<span style=\"font-weight: bold\"> mean_amount  </span>┃<span style=\"font-weight: bold\"> median_amount </span>┃\n┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>         │\n├───────────────┼───────────────┼─────────────┼──────────────┼──────────────┼───────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">50-100       </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2663933</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">155426540</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">58.344763</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">10-50        </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8115403</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">187666251</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">23.124699</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3636287</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">637353634</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">175.275943</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">150</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">&lt;10          </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2423728</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10080721</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.159180</span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">634677</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">334630687</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">527.245649</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">1000-5000    </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">684755</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1231394874</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1798.299938</span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1008</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">5000+        </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">44085</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1558371116</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35349.237065</span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10000</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">700821</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">123174568</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">175.757530</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">150</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">50-100       </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">304363</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">16184312</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">53.174374</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">10-50        </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">660787</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14411588</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21.809733</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└───────────────┴───────────────┴─────────────┴──────────────┴──────────────┴───────────────┘\n</pre>\n```\n:::\n:::\n\n\nOK, now let's do our timings.\n\nOne interesting thing to pay attention to here is the execution time for the following\ngroupby. Before, we could get away with lazy execution: because we only wanted to preview\nthe first few rows, we only had to compute the first few rows, so all our previews were\nvery fast.\n\nBut now, as soon as we do a groupby, we have to actually go through the whole dataset\nin order to compute the aggregate per group. So this is going to be slower. BUT,\nduckdb is still quite fast. It only takes milliseconds to groupby-agg all 20 million rows!\n\n::: {#fc3694c3 .cell execution_count=20}\n``` {.python .cell-code}\n%timeit summary_by(subset, [\"election_type\", \"amount_bucket\"]).execute()  # .execute() so we actually fetch the data\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n679 ms ± 11.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n```\n:::\n:::\n\n\nNow let's try the same thing in pandas:\n\n::: {#ab990661 .cell execution_count=21}\n``` {.python .cell-code}\n%timeit summary_by_pandas(pandas_subset, [\"election_type\", \"amount_bucket\"])\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n3.59 s ± 31.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n```\n:::\n:::\n\n\nIt takes about 4 seconds, which is about 10 times slower than duckdb.\nAt this scale, it again doesn't matter,\nbut you could imagine with a dataset much larger than this, it would matter.\n\nLet's also think about memory usage:\n\n::: {#03834f0b .cell execution_count=22}\n``` {.python .cell-code}\npandas_subset.memory_usage(deep=True).sum() / 1e9  # GB\n```\n\n::: {.cell-output .cell-output-display execution_count=22}\n```\n2.782586663\n```\n:::\n:::\n\n\nThe source dataframe is couple gigabytes, so probably during the groupby,\nthe peak memory usage is going to be a bit higher than this. You could use a profiler\nsuch as [FIL](https://github.com/pythonspeed/filprofiler) if you wanted an exact number,\nI was too lazy to use that here.\n\nAgain, this works on my laptop at this dataset size, but much larger than this and I'd\nstart having problems. Duckdb on the other hand is designed around working out of core\nso it should scale to datasets into the hundreds of gigabytes, much larger than your\ncomputer's RAM.\n\n### Back to analysis\n\nOK, let's plot the result of that groupby.\n\nSurprise! (Or maybe not...) Most donations are small. But most of the money comes\nfrom donations larger than $1000.\n\nWell if that's the case, why do politicians spend so much time soliciting small\ndonations? One explanation is that they can use the number of donations\nas a marketing pitch, to show how popular they are, and thus how viable of a\ncandidate they are.\n\nThis also might explain whose interests are being served by our politicians.\n\n::: {#cf2c035e .cell execution_count=23}\n``` {.python .cell-code}\nimport altair as alt\n\n# Do some bookkeeping so the buckets are displayed smallest to largest on the charts\nbucket_col = alt.Column(\"amount_bucket:N\", sort=labels)\n\nn_by_bucket = (\n    alt.Chart(by_type_and_bucket.execute())\n    .mark_bar()\n    .encode(\n        x=bucket_col,\n        y=\"n_donations:Q\",\n        color=\"election_type:N\",\n    )\n)\ntotal_by_bucket = (\n    alt.Chart(by_type_and_bucket.execute())\n    .mark_bar()\n    .encode(\n        x=bucket_col,\n        y=\"total_amount:Q\",\n        color=\"election_type:N\",\n    )\n)\nn_by_bucket | total_by_bucket\n```\n\n::: {.cell-output .cell-output-display execution_count=23}\n```{=html}\n\n<style>\n  #altair-viz-86646cc3a2964613b7509f2dcc47f742.vega-embed {\n    width: 100%;\n    display: flex;\n  }\n\n  #altair-viz-86646cc3a2964613b7509f2dcc47f742.vega-embed details,\n  #altair-viz-86646cc3a2964613b7509f2dcc47f742.vega-embed details summary {\n    position: relative;\n  }\n</style>\n<div id=\"altair-viz-86646cc3a2964613b7509f2dcc47f742\"></div>\n<script type=\"text/javascript\">\n  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n  (function(spec, embedOpt){\n    let outputDiv = document.currentScript.previousElementSibling;\n    if (outputDiv.id !== \"altair-viz-86646cc3a2964613b7509f2dcc47f742\") {\n      outputDiv = document.getElementById(\"altair-viz-86646cc3a2964613b7509f2dcc47f742\");\n    }\n    const paths = {\n      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.8.0?noext\",\n      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n    };\n\n    function maybeLoadScript(lib, version) {\n      var key = `${lib.replace(\"-\", \"\")}_version`;\n      return (VEGA_DEBUG[key] == version) ?\n        Promise.resolve(paths[lib]) :\n        new Promise(function(resolve, reject) {\n          var s = document.createElement('script');\n          document.getElementsByTagName(\"head\")[0].appendChild(s);\n          s.async = true;\n          s.onload = () => {\n            VEGA_DEBUG[key] = version;\n            return resolve(paths[lib]);\n          };\n          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n          s.src = paths[lib];\n        });\n    }\n\n    function showError(err) {\n      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n      throw err;\n    }\n\n    function displayChart(vegaEmbed) {\n      vegaEmbed(outputDiv, spec, embedOpt)\n        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n    }\n\n    if(typeof define === \"function\" && define.amd) {\n      requirejs.config({paths});\n      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n    } else {\n      maybeLoadScript(\"vega\", \"5\")\n        .then(() => maybeLoadScript(\"vega-lite\", \"5.8.0\"))\n        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n        .catch(showError)\n        .then(() => displayChart(vegaEmbed));\n    }\n  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"hconcat\": [{\"data\": {\"name\": \"data-7df486c18103705f447413da2d200c84\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"color\": {\"field\": \"election_type\", \"type\": \"nominal\"}, \"x\": {\"field\": \"amount_bucket\", \"sort\": [\"<10\", \"10-50\", \"50-100\", \"100-500\", \"500-1000\", \"1000-5000\", \"5000+\"], \"type\": \"nominal\"}, \"y\": {\"field\": \"n_donations\", \"type\": \"quantitative\"}}}, {\"data\": {\"name\": \"data-334b351fcf36268adfd9e34181060327\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"color\": {\"field\": \"election_type\", \"type\": \"nominal\"}, \"x\": {\"field\": \"amount_bucket\", \"sort\": [\"<10\", \"10-50\", \"50-100\", \"100-500\", \"500-1000\", \"1000-5000\", \"5000+\"], \"type\": \"nominal\"}, \"y\": {\"field\": \"total_amount\", \"type\": \"quantitative\"}}}], \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.8.0.json\", \"datasets\": {\"data-7df486c18103705f447413da2d200c84\": [{\"election_type\": \"primary\", \"amount_bucket\": \"<10\", \"n_donations\": 2423728, \"total_amount\": 10080721, \"mean_amount\": 4.159179990493983, \"median_amount\": 5}, {\"election_type\": \"primary\", \"amount_bucket\": \"10-50\", \"n_donations\": 8115403, \"total_amount\": 187666251, \"mean_amount\": 23.12469892129818, \"median_amount\": 25}, {\"election_type\": \"primary\", \"amount_bucket\": \"100-500\", \"n_donations\": 3636287, \"total_amount\": 637353634, \"mean_amount\": 175.27594329050484, \"median_amount\": 150}, {\"election_type\": \"primary\", \"amount_bucket\": \"50-100\", \"n_donations\": 2663933, \"total_amount\": 155426540, \"mean_amount\": 58.34476317535013, \"median_amount\": 50}, {\"election_type\": \"primary\", \"amount_bucket\": \"5000+\", \"n_donations\": 44085, \"total_amount\": 1558371116, \"mean_amount\": 35349.237064761255, \"median_amount\": 10000}, {\"election_type\": \"primary\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 684755, \"total_amount\": 1231394874, \"mean_amount\": 1798.2999379340056, \"median_amount\": 1008}, {\"election_type\": \"general\", \"amount_bucket\": \"100-500\", \"n_donations\": 700821, \"total_amount\": 123174568, \"mean_amount\": 175.75753009684357, \"median_amount\": 149}, {\"election_type\": \"primary\", \"amount_bucket\": \"500-1000\", \"n_donations\": 634677, \"total_amount\": 334630687, \"mean_amount\": 527.2456493618014, \"median_amount\": 500}, {\"election_type\": \"general\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 246101, \"total_amount\": 460025242, \"mean_amount\": 1869.2538510611496, \"median_amount\": 1959}, {\"election_type\": \"general\", \"amount_bucket\": \"10-50\", \"n_donations\": 660787, \"total_amount\": 14411588, \"mean_amount\": 21.809732939661345, \"median_amount\": 25}, {\"election_type\": \"general\", \"amount_bucket\": \"50-100\", \"n_donations\": 304363, \"total_amount\": 16184312, \"mean_amount\": 53.174374020495264, \"median_amount\": 50}, {\"election_type\": \"general\", \"amount_bucket\": \"500-1000\", \"n_donations\": 174182, \"total_amount\": 91015697, \"mean_amount\": 522.5321617618354, \"median_amount\": 500}, {\"election_type\": \"general\", \"amount_bucket\": \"<10\", \"n_donations\": 115873, \"total_amount\": 536742, \"mean_amount\": 4.632157620843509, \"median_amount\": 5}, {\"election_type\": \"runoff\", \"amount_bucket\": \"10-50\", \"n_donations\": 20166, \"total_amount\": 461107, \"mean_amount\": 22.865565803828225, \"median_amount\": 25}, {\"election_type\": \"runoff\", \"amount_bucket\": \"50-100\", \"n_donations\": 11578, \"total_amount\": 585827, \"mean_amount\": 50.59828986007946, \"median_amount\": 50}, {\"election_type\": \"runoff\", \"amount_bucket\": \"100-500\", \"n_donations\": 18193, \"total_amount\": 3088289, \"mean_amount\": 169.75149782883526, \"median_amount\": 100}, {\"election_type\": \"runoff\", \"amount_bucket\": \"500-1000\", \"n_donations\": 4117, \"total_amount\": 2110393, \"mean_amount\": 512.6045664318679, \"median_amount\": 500}, {\"election_type\": \"runoff\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 5196, \"total_amount\": 9601993, \"mean_amount\": 1847.958622016936, \"median_amount\": 1894}, {\"election_type\": \"runoff\", \"amount_bucket\": \"<10\", \"n_donations\": 10191, \"total_amount\": 49621, \"mean_amount\": 4.869100186439015, \"median_amount\": 5}, {\"election_type\": \"special\", \"amount_bucket\": \"100-500\", \"n_donations\": 34497, \"total_amount\": 5943498, \"mean_amount\": 172.29028611183583, \"median_amount\": 117}, {\"election_type\": \"general\", \"amount_bucket\": \"5000+\", \"n_donations\": 3125, \"total_amount\": 44496373, \"mean_amount\": 14238.83936, \"median_amount\": 7541}, {\"election_type\": \"special\", \"amount_bucket\": \"5000+\", \"n_donations\": 129, \"total_amount\": 788712, \"mean_amount\": 6114.046511627907, \"median_amount\": 5400}, {\"election_type\": \"other\", \"amount_bucket\": \"500-1000\", \"n_donations\": 119, \"total_amount\": 62535, \"mean_amount\": 525.5042016806723, \"median_amount\": 500}, {\"election_type\": \"other\", \"amount_bucket\": \"100-500\", \"n_donations\": 630, \"total_amount\": 117988, \"mean_amount\": 187.2825396825397, \"median_amount\": 192}, {\"election_type\": \"other\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 235, \"total_amount\": 548212, \"mean_amount\": 2332.817021276596, \"median_amount\": 2633}, {\"election_type\": \"other\", \"amount_bucket\": \"50-100\", \"n_donations\": 451, \"total_amount\": 27149, \"mean_amount\": 60.19733924611973, \"median_amount\": 50}, {\"election_type\": \"other\", \"amount_bucket\": \"10-50\", \"n_donations\": 2644, \"total_amount\": 64297, \"mean_amount\": 24.318078668683814, \"median_amount\": 23}, {\"election_type\": \"convention\", \"amount_bucket\": \"100-500\", \"n_donations\": 6350, \"total_amount\": 1097843, \"mean_amount\": 172.88866141732282, \"median_amount\": 138}, {\"election_type\": \"convention\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 2822, \"total_amount\": 4977314, \"mean_amount\": 1763.7540751240256, \"median_amount\": 1434}, {\"election_type\": \"convention\", \"amount_bucket\": \"10-50\", \"n_donations\": 6848, \"total_amount\": 141604, \"mean_amount\": 20.678154205607477, \"median_amount\": 25}, {\"election_type\": \"convention\", \"amount_bucket\": \"500-1000\", \"n_donations\": 1824, \"total_amount\": 945321, \"mean_amount\": 518.2680921052631, \"median_amount\": 500}, {\"election_type\": \"convention\", \"amount_bucket\": \"50-100\", \"n_donations\": 2966, \"total_amount\": 153281, \"mean_amount\": 51.67936614969656, \"median_amount\": 50}, {\"election_type\": \"convention\", \"amount_bucket\": \"<10\", \"n_donations\": 945, \"total_amount\": 4660, \"mean_amount\": 4.931216931216931, \"median_amount\": 5}, {\"election_type\": \"other\", \"amount_bucket\": \"<10\", \"n_donations\": 10993, \"total_amount\": 25816, \"mean_amount\": 2.3484035295187846, \"median_amount\": 1}, {\"election_type\": \"special\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 7935, \"total_amount\": 13493154, \"mean_amount\": 1700.4604914933836, \"median_amount\": 1001}, {\"election_type\": \"special\", \"amount_bucket\": \"50-100\", \"n_donations\": 22859, \"total_amount\": 1177660, \"mean_amount\": 51.518439126820944, \"median_amount\": 50}, {\"election_type\": \"special\", \"amount_bucket\": \"<10\", \"n_donations\": 25115, \"total_amount\": 122898, \"mean_amount\": 4.893410312562214, \"median_amount\": 5}, {\"election_type\": \"special\", \"amount_bucket\": \"10-50\", \"n_donations\": 51066, \"total_amount\": 1134616, \"mean_amount\": 22.21861904202405, \"median_amount\": 25}, {\"election_type\": \"special\", \"amount_bucket\": \"500-1000\", \"n_donations\": 7811, \"total_amount\": 4003293, \"mean_amount\": 512.5199078223019, \"median_amount\": 500}, {\"election_type\": \"runoff\", \"amount_bucket\": \"5000+\", \"n_donations\": 37, \"total_amount\": 211400, \"mean_amount\": 5713.513513513513, \"median_amount\": 5400}, {\"election_type\": null, \"amount_bucket\": \"1000-5000\", \"n_donations\": 116, \"total_amount\": 228657, \"mean_amount\": 1971.1810344827586, \"median_amount\": 1300}, {\"election_type\": \"recount\", \"amount_bucket\": \"5000+\", \"n_donations\": 26, \"total_amount\": 1888024, \"mean_amount\": 72616.30769230769, \"median_amount\": 101450}, {\"election_type\": null, \"amount_bucket\": \"5000+\", \"n_donations\": 48, \"total_amount\": 1622455, \"mean_amount\": 33801.145833333336, \"median_amount\": 21731}, {\"election_type\": \"recount\", \"amount_bucket\": \"100-500\", \"n_donations\": 2232, \"total_amount\": 413753, \"mean_amount\": 185.37320788530465, \"median_amount\": 200}, {\"election_type\": \"recount\", \"amount_bucket\": \"10-50\", \"n_donations\": 883, \"total_amount\": 20860, \"mean_amount\": 23.62400906002265, \"median_amount\": 25}, {\"election_type\": \"recount\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 589, \"total_amount\": 1113150, \"mean_amount\": 1889.8981324278438, \"median_amount\": 1965}, {\"election_type\": \"recount\", \"amount_bucket\": \"500-1000\", \"n_donations\": 494, \"total_amount\": 250960, \"mean_amount\": 508.0161943319838, \"median_amount\": 500}, {\"election_type\": \"recount\", \"amount_bucket\": \"50-100\", \"n_donations\": 712, \"total_amount\": 38450, \"mean_amount\": 54.002808988764045, \"median_amount\": 50}, {\"election_type\": \"convention\", \"amount_bucket\": \"5000+\", \"n_donations\": 219, \"total_amount\": 1590300, \"mean_amount\": 7261.643835616438, \"median_amount\": 8100}, {\"election_type\": null, \"amount_bucket\": \"100-500\", \"n_donations\": 195, \"total_amount\": 46746, \"mean_amount\": 239.72307692307692, \"median_amount\": 250}, {\"election_type\": null, \"amount_bucket\": \"500-1000\", \"n_donations\": 89, \"total_amount\": 48290, \"mean_amount\": 542.5842696629213, \"median_amount\": 500}, {\"election_type\": null, \"amount_bucket\": \"50-100\", \"n_donations\": 36, \"total_amount\": 1880, \"mean_amount\": 52.22222222222222, \"median_amount\": 50}, {\"election_type\": \"recount\", \"amount_bucket\": \"<10\", \"n_donations\": 110, \"total_amount\": 569, \"mean_amount\": 5.172727272727273, \"median_amount\": 5}, {\"election_type\": null, \"amount_bucket\": \"10-50\", \"n_donations\": 151, \"total_amount\": 3167, \"mean_amount\": 20.973509933774835, \"median_amount\": 25}, {\"election_type\": null, \"amount_bucket\": \"<10\", \"n_donations\": 24, \"total_amount\": 108, \"mean_amount\": 4.5, \"median_amount\": 5}, {\"election_type\": \"other\", \"amount_bucket\": \"5000+\", \"n_donations\": 48, \"total_amount\": 1901300, \"mean_amount\": 39610.416666666664, \"median_amount\": 16950}], \"data-334b351fcf36268adfd9e34181060327\": [{\"election_type\": \"primary\", \"amount_bucket\": \"<10\", \"n_donations\": 2423728, \"total_amount\": 10080721, \"mean_amount\": 4.159179990493983, \"median_amount\": 5}, {\"election_type\": \"primary\", \"amount_bucket\": \"10-50\", \"n_donations\": 8115403, \"total_amount\": 187666251, \"mean_amount\": 23.12469892129818, \"median_amount\": 25}, {\"election_type\": \"primary\", \"amount_bucket\": \"100-500\", \"n_donations\": 3636287, \"total_amount\": 637353634, \"mean_amount\": 175.27594329050484, \"median_amount\": 150}, {\"election_type\": \"primary\", \"amount_bucket\": \"50-100\", \"n_donations\": 2663933, \"total_amount\": 155426540, \"mean_amount\": 58.34476317535013, \"median_amount\": 50}, {\"election_type\": \"primary\", \"amount_bucket\": \"5000+\", \"n_donations\": 44085, \"total_amount\": 1558371116, \"mean_amount\": 35349.237064761255, \"median_amount\": 10000}, {\"election_type\": \"primary\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 684755, \"total_amount\": 1231394874, \"mean_amount\": 1798.2999379340056, \"median_amount\": 1005}, {\"election_type\": \"general\", \"amount_bucket\": \"100-500\", \"n_donations\": 700821, \"total_amount\": 123174568, \"mean_amount\": 175.75753009684357, \"median_amount\": 150}, {\"election_type\": \"primary\", \"amount_bucket\": \"500-1000\", \"n_donations\": 634677, \"total_amount\": 334630687, \"mean_amount\": 527.2456493618014, \"median_amount\": 500}, {\"election_type\": \"general\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 246101, \"total_amount\": 460025242, \"mean_amount\": 1869.2538510611496, \"median_amount\": 1946}, {\"election_type\": \"general\", \"amount_bucket\": \"10-50\", \"n_donations\": 660787, \"total_amount\": 14411588, \"mean_amount\": 21.809732939661345, \"median_amount\": 25}, {\"election_type\": \"general\", \"amount_bucket\": \"50-100\", \"n_donations\": 304363, \"total_amount\": 16184312, \"mean_amount\": 53.174374020495264, \"median_amount\": 50}, {\"election_type\": \"general\", \"amount_bucket\": \"500-1000\", \"n_donations\": 174182, \"total_amount\": 91015697, \"mean_amount\": 522.5321617618354, \"median_amount\": 500}, {\"election_type\": \"general\", \"amount_bucket\": \"<10\", \"n_donations\": 115873, \"total_amount\": 536742, \"mean_amount\": 4.632157620843509, \"median_amount\": 5}, {\"election_type\": \"runoff\", \"amount_bucket\": \"10-50\", \"n_donations\": 20166, \"total_amount\": 461107, \"mean_amount\": 22.865565803828225, \"median_amount\": 25}, {\"election_type\": \"runoff\", \"amount_bucket\": \"50-100\", \"n_donations\": 11578, \"total_amount\": 585827, \"mean_amount\": 50.59828986007946, \"median_amount\": 50}, {\"election_type\": \"runoff\", \"amount_bucket\": \"100-500\", \"n_donations\": 18193, \"total_amount\": 3088289, \"mean_amount\": 169.75149782883526, \"median_amount\": 100}, {\"election_type\": \"runoff\", \"amount_bucket\": \"500-1000\", \"n_donations\": 4117, \"total_amount\": 2110393, \"mean_amount\": 512.6045664318679, \"median_amount\": 500}, {\"election_type\": \"runoff\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 5196, \"total_amount\": 9601993, \"mean_amount\": 1847.958622016936, \"median_amount\": 1901}, {\"election_type\": \"runoff\", \"amount_bucket\": \"<10\", \"n_donations\": 10191, \"total_amount\": 49621, \"mean_amount\": 4.869100186439015, \"median_amount\": 5}, {\"election_type\": \"special\", \"amount_bucket\": \"100-500\", \"n_donations\": 34497, \"total_amount\": 5943498, \"mean_amount\": 172.29028611183583, \"median_amount\": 118}, {\"election_type\": \"general\", \"amount_bucket\": \"5000+\", \"n_donations\": 3125, \"total_amount\": 44496373, \"mean_amount\": 14238.83936, \"median_amount\": 7528}, {\"election_type\": \"special\", \"amount_bucket\": \"5000+\", \"n_donations\": 129, \"total_amount\": 788712, \"mean_amount\": 6114.046511627907, \"median_amount\": 5400}, {\"election_type\": \"special\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 7935, \"total_amount\": 13493154, \"mean_amount\": 1700.4604914933836, \"median_amount\": 1000}, {\"election_type\": \"convention\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 2822, \"total_amount\": 4977314, \"mean_amount\": 1763.7540751240256, \"median_amount\": 1437}, {\"election_type\": \"convention\", \"amount_bucket\": \"100-500\", \"n_donations\": 6350, \"total_amount\": 1097843, \"mean_amount\": 172.88866141732282, \"median_amount\": 139}, {\"election_type\": \"convention\", \"amount_bucket\": \"500-1000\", \"n_donations\": 1824, \"total_amount\": 945321, \"mean_amount\": 518.2680921052631, \"median_amount\": 500}, {\"election_type\": \"special\", \"amount_bucket\": \"10-50\", \"n_donations\": 51066, \"total_amount\": 1134616, \"mean_amount\": 22.21861904202405, \"median_amount\": 25}, {\"election_type\": \"special\", \"amount_bucket\": \"50-100\", \"n_donations\": 22859, \"total_amount\": 1177660, \"mean_amount\": 51.518439126820944, \"median_amount\": 50}, {\"election_type\": \"special\", \"amount_bucket\": \"500-1000\", \"n_donations\": 7811, \"total_amount\": 4003293, \"mean_amount\": 512.5199078223019, \"median_amount\": 500}, {\"election_type\": \"special\", \"amount_bucket\": \"<10\", \"n_donations\": 25115, \"total_amount\": 122898, \"mean_amount\": 4.893410312562214, \"median_amount\": 5}, {\"election_type\": null, \"amount_bucket\": \"1000-5000\", \"n_donations\": 116, \"total_amount\": 228657, \"mean_amount\": 1971.1810344827586, \"median_amount\": 1300}, {\"election_type\": null, \"amount_bucket\": \"5000+\", \"n_donations\": 48, \"total_amount\": 1622455, \"mean_amount\": 33801.145833333336, \"median_amount\": 21731}, {\"election_type\": \"convention\", \"amount_bucket\": \"10-50\", \"n_donations\": 6848, \"total_amount\": 141604, \"mean_amount\": 20.678154205607477, \"median_amount\": 25}, {\"election_type\": \"convention\", \"amount_bucket\": \"<10\", \"n_donations\": 945, \"total_amount\": 4660, \"mean_amount\": 4.931216931216931, \"median_amount\": 5}, {\"election_type\": \"convention\", \"amount_bucket\": \"50-100\", \"n_donations\": 2966, \"total_amount\": 153281, \"mean_amount\": 51.67936614969656, \"median_amount\": 50}, {\"election_type\": \"convention\", \"amount_bucket\": \"5000+\", \"n_donations\": 219, \"total_amount\": 1590300, \"mean_amount\": 7261.643835616438, \"median_amount\": 8100}, {\"election_type\": \"recount\", \"amount_bucket\": \"100-500\", \"n_donations\": 2232, \"total_amount\": 413753, \"mean_amount\": 185.37320788530465, \"median_amount\": 200}, {\"election_type\": \"recount\", \"amount_bucket\": \"10-50\", \"n_donations\": 883, \"total_amount\": 20860, \"mean_amount\": 23.62400906002265, \"median_amount\": 25}, {\"election_type\": \"recount\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 589, \"total_amount\": 1113150, \"mean_amount\": 1889.8981324278438, \"median_amount\": 1965}, {\"election_type\": \"recount\", \"amount_bucket\": \"500-1000\", \"n_donations\": 494, \"total_amount\": 250960, \"mean_amount\": 508.0161943319838, \"median_amount\": 500}, {\"election_type\": \"recount\", \"amount_bucket\": \"50-100\", \"n_donations\": 712, \"total_amount\": 38450, \"mean_amount\": 54.002808988764045, \"median_amount\": 50}, {\"election_type\": \"recount\", \"amount_bucket\": \"<10\", \"n_donations\": 110, \"total_amount\": 569, \"mean_amount\": 5.172727272727273, \"median_amount\": 5}, {\"election_type\": null, \"amount_bucket\": \"100-500\", \"n_donations\": 195, \"total_amount\": 46746, \"mean_amount\": 239.72307692307692, \"median_amount\": 250}, {\"election_type\": null, \"amount_bucket\": \"500-1000\", \"n_donations\": 89, \"total_amount\": 48290, \"mean_amount\": 542.5842696629213, \"median_amount\": 500}, {\"election_type\": \"other\", \"amount_bucket\": \"10-50\", \"n_donations\": 2644, \"total_amount\": 64297, \"mean_amount\": 24.318078668683814, \"median_amount\": 23}, {\"election_type\": \"other\", \"amount_bucket\": \"50-100\", \"n_donations\": 451, \"total_amount\": 27149, \"mean_amount\": 60.19733924611973, \"median_amount\": 50}, {\"election_type\": \"other\", \"amount_bucket\": \"100-500\", \"n_donations\": 630, \"total_amount\": 117988, \"mean_amount\": 187.2825396825397, \"median_amount\": 192}, {\"election_type\": \"other\", \"amount_bucket\": \"<10\", \"n_donations\": 10993, \"total_amount\": 25816, \"mean_amount\": 2.3484035295187846, \"median_amount\": 1}, {\"election_type\": \"other\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 235, \"total_amount\": 548212, \"mean_amount\": 2332.817021276596, \"median_amount\": 2633}, {\"election_type\": \"recount\", \"amount_bucket\": \"5000+\", \"n_donations\": 26, \"total_amount\": 1888024, \"mean_amount\": 72616.30769230769, \"median_amount\": 101450}, {\"election_type\": null, \"amount_bucket\": \"<10\", \"n_donations\": 24, \"total_amount\": 108, \"mean_amount\": 4.5, \"median_amount\": 5}, {\"election_type\": null, \"amount_bucket\": \"10-50\", \"n_donations\": 151, \"total_amount\": 3167, \"mean_amount\": 20.973509933774835, \"median_amount\": 25}, {\"election_type\": \"other\", \"amount_bucket\": \"500-1000\", \"n_donations\": 119, \"total_amount\": 62535, \"mean_amount\": 525.5042016806723, \"median_amount\": 500}, {\"election_type\": \"runoff\", \"amount_bucket\": \"5000+\", \"n_donations\": 37, \"total_amount\": 211400, \"mean_amount\": 5713.513513513513, \"median_amount\": 5400}, {\"election_type\": null, \"amount_bucket\": \"50-100\", \"n_donations\": 36, \"total_amount\": 1880, \"mean_amount\": 52.22222222222222, \"median_amount\": 50}, {\"election_type\": \"other\", \"amount_bucket\": \"5000+\", \"n_donations\": 48, \"total_amount\": 1901300, \"mean_amount\": 39610.416666666664, \"median_amount\": 16950}]}}, {\"mode\": \"vega-lite\"});\n</script>\n```\n:::\n:::\n\n\n### By election stage\n\nLet's look at how donations break down by election stage. Do people donate\ndifferently for primary elections vs general elections?\n\nLet's ignore everything but primary and general elections, since they are the\nmost common, and arguably the most important.\n\n::: {#92651642 .cell execution_count=24}\n``` {.python .cell-code}\ngb2 = by_type_and_bucket[_.election_type.isin((\"primary\", \"general\"))]\nn_donations_per_election_type = _.n_donations.sum().over(group_by=\"election_type\")\nfrac = _.n_donations / n_donations_per_election_type\ngb2 = gb2.mutate(frac_n_donations_per_election_type=frac)\ngb2\n```\n\n::: {.cell-output .cell-output-display execution_count=24}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> election_type </span>┃<span style=\"font-weight: bold\"> amount_bucket </span>┃<span style=\"font-weight: bold\"> n_donations </span>┃<span style=\"font-weight: bold\"> total_amount </span>┃<span style=\"font-weight: bold\"> mean_amount  </span>┃<span style=\"font-weight: bold\"> median_amount </span>┃<span style=\"font-weight: bold\"> frac_n_donations_per_election_type </span>┃\n┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>                            │\n├───────────────┼───────────────┼─────────────┼──────────────┼──────────────┼───────────────┼────────────────────────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">10-50        </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8115403</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">187666251</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">23.124699</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.445831</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">&lt;10          </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2423728</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10080721</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.159180</span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.133151</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3636287</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">637353634</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">175.275943</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">150</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.199765</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">50-100       </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2663933</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">155426540</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">58.344763</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.146347</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">634677</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">334630687</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">527.245649</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.034867</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">1000-5000    </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">684755</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1231394874</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1798.299938</span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1008</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.037618</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">5000+        </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">44085</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1558371116</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35349.237065</span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10000</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.002422</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">50-100       </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">304363</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">16184312</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">53.174374</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.138017</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">700821</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">123174568</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">175.757530</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">150</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.317796</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">174182</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">91015697</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">522.532162</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.078985</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└───────────────┴───────────────┴─────────────┴──────────────┴──────────────┴───────────────┴────────────────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\nIt looks like primary elections get a larger proportion of small donations.\n\n::: {#fd42d9bf .cell execution_count=25}\n``` {.python .cell-code}\nalt.Chart(gb2.execute()).mark_bar().encode(\n    x=\"election_type:O\",\n    y=\"frac_n_donations_per_election_type:Q\",\n    color=bucket_col,\n)\n```\n\n::: {.cell-output .cell-output-display execution_count=25}\n```{=html}\n\n<style>\n  #altair-viz-d70f3d08fb7d48c3b9c5b7d4e4e4b735.vega-embed {\n    width: 100%;\n    display: flex;\n  }\n\n  #altair-viz-d70f3d08fb7d48c3b9c5b7d4e4e4b735.vega-embed details,\n  #altair-viz-d70f3d08fb7d48c3b9c5b7d4e4e4b735.vega-embed details summary {\n    position: relative;\n  }\n</style>\n<div id=\"altair-viz-d70f3d08fb7d48c3b9c5b7d4e4e4b735\"></div>\n<script type=\"text/javascript\">\n  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n  (function(spec, embedOpt){\n    let outputDiv = document.currentScript.previousElementSibling;\n    if (outputDiv.id !== \"altair-viz-d70f3d08fb7d48c3b9c5b7d4e4e4b735\") {\n      outputDiv = document.getElementById(\"altair-viz-d70f3d08fb7d48c3b9c5b7d4e4e4b735\");\n    }\n    const paths = {\n      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.8.0?noext\",\n      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n    };\n\n    function maybeLoadScript(lib, version) {\n      var key = `${lib.replace(\"-\", \"\")}_version`;\n      return (VEGA_DEBUG[key] == version) ?\n        Promise.resolve(paths[lib]) :\n        new Promise(function(resolve, reject) {\n          var s = document.createElement('script');\n          document.getElementsByTagName(\"head\")[0].appendChild(s);\n          s.async = true;\n          s.onload = () => {\n            VEGA_DEBUG[key] = version;\n            return resolve(paths[lib]);\n          };\n          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n          s.src = paths[lib];\n        });\n    }\n\n    function showError(err) {\n      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n      throw err;\n    }\n\n    function displayChart(vegaEmbed) {\n      vegaEmbed(outputDiv, spec, embedOpt)\n        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n    }\n\n    if(typeof define === \"function\" && define.amd) {\n      requirejs.config({paths});\n      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n    } else {\n      maybeLoadScript(\"vega\", \"5\")\n        .then(() => maybeLoadScript(\"vega-lite\", \"5.8.0\"))\n        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n        .catch(showError)\n        .then(() => displayChart(vegaEmbed));\n    }\n  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-a720be181ccb86f83b23a767dba558e0\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"color\": {\"field\": \"amount_bucket\", \"sort\": [\"<10\", \"10-50\", \"50-100\", \"100-500\", \"500-1000\", \"1000-5000\", \"5000+\"], \"type\": \"nominal\"}, \"x\": {\"field\": \"election_type\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"frac_n_donations_per_election_type\", \"type\": \"quantitative\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.8.0.json\", \"datasets\": {\"data-a720be181ccb86f83b23a767dba558e0\": [{\"election_type\": \"primary\", \"amount_bucket\": \"<10\", \"n_donations\": 2423728, \"total_amount\": 10080721, \"mean_amount\": 4.159179990493983, \"median_amount\": 5, \"frac_n_donations_per_election_type\": 0.13315088589336582}, {\"election_type\": \"primary\", \"amount_bucket\": \"10-50\", \"n_donations\": 8115403, \"total_amount\": 187666251, \"mean_amount\": 23.12469892129818, \"median_amount\": 25, \"frac_n_donations_per_election_type\": 0.44583100860809405}, {\"election_type\": \"primary\", \"amount_bucket\": \"50-100\", \"n_donations\": 2663933, \"total_amount\": 155426540, \"mean_amount\": 58.34476317535013, \"median_amount\": 50, \"frac_n_donations_per_election_type\": 0.14634688335925966}, {\"election_type\": \"primary\", \"amount_bucket\": \"100-500\", \"n_donations\": 3636287, \"total_amount\": 637353634, \"mean_amount\": 175.27594329050484, \"median_amount\": 150, \"frac_n_donations_per_election_type\": 0.1997645096366133}, {\"election_type\": \"primary\", \"amount_bucket\": \"500-1000\", \"n_donations\": 634677, \"total_amount\": 334630687, \"mean_amount\": 527.2456493618014, \"median_amount\": 500, \"frac_n_donations_per_election_type\": 0.03486686823197312}, {\"election_type\": \"primary\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 684755, \"total_amount\": 1231394874, \"mean_amount\": 1798.2999379340056, \"median_amount\": 1008, \"frac_n_donations_per_election_type\": 0.037617973167744775}, {\"election_type\": \"primary\", \"amount_bucket\": \"5000+\", \"n_donations\": 44085, \"total_amount\": 1558371116, \"mean_amount\": 35349.237064761255, \"median_amount\": 10000, \"frac_n_donations_per_election_type\": 0.0024218711029492714}, {\"election_type\": \"general\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 246101, \"total_amount\": 460025242, \"mean_amount\": 1869.2538510611496, \"median_amount\": 1956, \"frac_n_donations_per_election_type\": 0.1115976768187944}, {\"election_type\": \"general\", \"amount_bucket\": \"100-500\", \"n_donations\": 700821, \"total_amount\": 123174568, \"mean_amount\": 175.75753009684357, \"median_amount\": 150, \"frac_n_donations_per_election_type\": 0.3177963334802553}, {\"election_type\": \"general\", \"amount_bucket\": \"50-100\", \"n_donations\": 304363, \"total_amount\": 16184312, \"mean_amount\": 53.174374020495264, \"median_amount\": 50, \"frac_n_donations_per_election_type\": 0.13801733316645898}, {\"election_type\": \"general\", \"amount_bucket\": \"<10\", \"n_donations\": 115873, \"total_amount\": 536742, \"mean_amount\": 4.632157620843509, \"median_amount\": 5, \"frac_n_donations_per_election_type\": 0.052544108337731925}, {\"election_type\": \"general\", \"amount_bucket\": \"500-1000\", \"n_donations\": 174182, \"total_amount\": 91015697, \"mean_amount\": 522.5321617618354, \"median_amount\": 500, \"frac_n_donations_per_election_type\": 0.0789850774423966}, {\"election_type\": \"general\", \"amount_bucket\": \"10-50\", \"n_donations\": 660787, \"total_amount\": 14411588, \"mean_amount\": 21.809732939661345, \"median_amount\": 25, \"frac_n_donations_per_election_type\": 0.2996423991453131}, {\"election_type\": \"general\", \"amount_bucket\": \"5000+\", \"n_donations\": 3125, \"total_amount\": 44496373, \"mean_amount\": 14238.83936, \"median_amount\": 7525, \"frac_n_donations_per_election_type\": 0.0014170716090496688}]}}, {\"mode\": \"vega-lite\"});\n</script>\n```\n:::\n:::\n\n\n### By recipient\n\nLet's look at the top players. Who gets the most donations?\n\nFar and away it is ActBlue, which acts as a conduit for donations to Democratic\ninterests.\n\nBeto O'Rourke is the top individual politician, hats off to him!\n\n::: {#e844f42e .cell execution_count=26}\n``` {.python .cell-code}\nby_recip = summary_by(featured, \"CMTE_NM\")\nby_recip\n```\n\n::: {.cell-output .cell-output-display execution_count=26}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CMTE_NM                                                                          </span>┃<span style=\"font-weight: bold\"> n_donations </span>┃<span style=\"font-weight: bold\"> total_amount </span>┃<span style=\"font-weight: bold\"> mean_amount </span>┃<span style=\"font-weight: bold\"> median_amount </span>┃\n┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                                                           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>         │\n├──────────────────────────────────────────────────────────────────────────────────┼─────────────┼──────────────┼─────────────┼───────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">EXELON CORPORATION POLITICAL ACTION COMMITTEE (EXELON PAC)                      </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13250</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1939503</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">146.377585</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">118</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">ARCHER DANIELS MIDLAND COMPANY-ADM PAC                                          </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4460</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">275807</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">61.840135</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">PFIZER INC. PAC                                                                 </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">46900</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1948689</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">41.549872</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">SUEZ WATER INC. FEDERAL PAC                                                     </span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">16873</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">156.231481</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">120</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">INTERNATIONAL WAREHOUSE LOGISTICS ASSOCIATION PAC                               </span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">90</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">132200</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1468.888889</span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1000</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">BAKERY, CONFECTIONERY, TOBACCO WORKERS AND GRAIN MILLERS INTERNATIONAL UNION PAC</span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">387</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">19091</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">49.330749</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">30</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">UNION PACIFIC CORP. FUND FOR EFFECTIVE GOVERNMENT                               </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">16118</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2436963</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">151.195123</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">NATIONAL ASSOCIATION OF REALTORS POLITICAL ACTION COMMITTEE                     </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">24277</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5492063</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">226.224945</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">154</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">AMERICAN FINANCIAL SERVICES ASSOCIATION PAC                                     </span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">690</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">685839</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">993.969565</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">65</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">WEYERHAEUSER COMPANY POLITICAL ACTION COMMITTEE                                 </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5512</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">343244</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">62.272134</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">30</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                                                                │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└──────────────────────────────────────────────────────────────────────────────────┴─────────────┴──────────────┴─────────────┴───────────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#a0c1efd8 .cell execution_count=27}\n``` {.python .cell-code}\ntop_recip = by_recip.order_by(ibis.desc(\"n_donations\")).head(10)\nalt.Chart(top_recip.execute()).mark_bar().encode(\n    x=alt.X(\"CMTE_NM:O\", sort=\"-y\"),\n    y=\"n_donations:Q\",\n)\n```\n\n::: {.cell-output .cell-output-display execution_count=27}\n```{=html}\n\n<style>\n  #altair-viz-59298bf7a2fb45ba8365f4fc9ef956b9.vega-embed {\n    width: 100%;\n    display: flex;\n  }\n\n  #altair-viz-59298bf7a2fb45ba8365f4fc9ef956b9.vega-embed details,\n  #altair-viz-59298bf7a2fb45ba8365f4fc9ef956b9.vega-embed details summary {\n    position: relative;\n  }\n</style>\n<div id=\"altair-viz-59298bf7a2fb45ba8365f4fc9ef956b9\"></div>\n<script type=\"text/javascript\">\n  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n  (function(spec, embedOpt){\n    let outputDiv = document.currentScript.previousElementSibling;\n    if (outputDiv.id !== \"altair-viz-59298bf7a2fb45ba8365f4fc9ef956b9\") {\n      outputDiv = document.getElementById(\"altair-viz-59298bf7a2fb45ba8365f4fc9ef956b9\");\n    }\n    const paths = {\n      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.8.0?noext\",\n      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n    };\n\n    function maybeLoadScript(lib, version) {\n      var key = `${lib.replace(\"-\", \"\")}_version`;\n      return (VEGA_DEBUG[key] == version) ?\n        Promise.resolve(paths[lib]) :\n        new Promise(function(resolve, reject) {\n          var s = document.createElement('script');\n          document.getElementsByTagName(\"head\")[0].appendChild(s);\n          s.async = true;\n          s.onload = () => {\n            VEGA_DEBUG[key] = version;\n            return resolve(paths[lib]);\n          };\n          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n          s.src = paths[lib];\n        });\n    }\n\n    function showError(err) {\n      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n      throw err;\n    }\n\n    function displayChart(vegaEmbed) {\n      vegaEmbed(outputDiv, spec, embedOpt)\n        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n    }\n\n    if(typeof define === \"function\" && define.amd) {\n      requirejs.config({paths});\n      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n    } else {\n      maybeLoadScript(\"vega\", \"5\")\n        .then(() => maybeLoadScript(\"vega-lite\", \"5.8.0\"))\n        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n        .catch(showError)\n        .then(() => displayChart(vegaEmbed));\n    }\n  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-ce10b3f5b7c7e35451245a008d469163\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"CMTE_NM\", \"sort\": \"-y\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"n_donations\", \"type\": \"quantitative\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.8.0.json\", \"datasets\": {\"data-ce10b3f5b7c7e35451245a008d469163\": [{\"CMTE_NM\": \"ACTBLUE\", \"n_donations\": 5820888, \"total_amount\": 693057213, \"mean_amount\": 119.06382892094814, \"median_amount\": 25}, {\"CMTE_NM\": \"DCCC\", \"n_donations\": 1315476, \"total_amount\": 124802082, \"mean_amount\": 94.87218466927561, \"median_amount\": 25}, {\"CMTE_NM\": \"REPUBLICAN NATIONAL COMMITTEE\", \"n_donations\": 570561, \"total_amount\": 131525422, \"mean_amount\": 230.5194746924518, \"median_amount\": 50}, {\"CMTE_NM\": \"END CITIZENS UNITED\", \"n_donations\": 489710, \"total_amount\": 13654987, \"mean_amount\": 27.8838230789651, \"median_amount\": 15}, {\"CMTE_NM\": \"DSCC\", \"n_donations\": 347493, \"total_amount\": 67844824, \"mean_amount\": 195.2408365060591, \"median_amount\": 35}, {\"CMTE_NM\": \"PROGRESSIVE TURNOUT PROJECT\", \"n_donations\": 313433, \"total_amount\": 9251647, \"mean_amount\": 29.517144014829327, \"median_amount\": 15}, {\"CMTE_NM\": \"DNC SERVICES CORP./DEM. NAT'L COMMITTEE\", \"n_donations\": 280264, \"total_amount\": 70156788, \"mean_amount\": 250.32393743042275, \"median_amount\": 50}, {\"CMTE_NM\": \"BETO FOR TEXAS\", \"n_donations\": 280027, \"total_amount\": 44914966, \"mean_amount\": 160.39512618426082, \"median_amount\": 50}, {\"CMTE_NM\": \"NRSC\", \"n_donations\": 203124, \"total_amount\": 55384644, \"mean_amount\": 272.66420511608675, \"median_amount\": 50}, {\"CMTE_NM\": \"NRCC\", \"n_donations\": 178176, \"total_amount\": 38646560, \"mean_amount\": 216.90104166666666, \"median_amount\": 50}]}}, {\"mode\": \"vega-lite\"});\n</script>\n```\n:::\n:::\n\n\n### By Location\n\nWhere are the largest donations coming from?\n\n::: {#3348eca1 .cell execution_count=28}\n``` {.python .cell-code}\nf2 = featured.mutate(loc=_.CITY + \", \" + _.STATE).drop(\"CITY\", \"STATE\")\nby_loc = summary_by(f2, \"loc\")\n# Drop the places with a small number of donations so we're\n# resistant to outliers for the mean\nby_loc = by_loc[_.n_donations > 1000]\nby_loc\n```\n\n::: {.cell-output .cell-output-display execution_count=28}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> loc              </span>┃<span style=\"font-weight: bold\"> n_donations </span>┃<span style=\"font-weight: bold\"> total_amount </span>┃<span style=\"font-weight: bold\"> mean_amount </span>┃<span style=\"font-weight: bold\"> median_amount </span>┃\n┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>         │\n├──────────────────┼─────────────┼──────────────┼─────────────┼───────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">DALLAS, TX      </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">154038</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">66558403</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">432.090802</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">58</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">PHILADELPHIA, PA</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">222938</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">36054977</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">161.726476</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">62</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">MALIBU, CA      </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11699</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4934763</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">421.810668</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">SANTEE, CA      </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2454</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201274</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">82.018745</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">26</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">WINNETKA, IL    </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8589</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5621809</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">654.535918</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">172</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">OREM, UT        </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2110</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">837475</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">396.907583</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">MESA, AZ        </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">22128</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1856636</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">83.904375</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">WAYZATA, MN     </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6488</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3326275</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512.681104</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">117</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">MINNETONKA, MN  </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5709</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1187881</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">208.071641</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">OJAI, CA        </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4496</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">926422</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">206.054715</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└──────────────────┴─────────────┴──────────────┴─────────────┴───────────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#95c93760 .cell execution_count=29}\n``` {.python .cell-code}\ndef top_by(col):\n    top = by_loc.order_by(ibis.desc(col)).head(10)\n    return (\n        alt.Chart(top.execute())\n        .mark_bar()\n        .encode(\n            x=alt.X('loc:O', sort=\"-y\"),\n            y=col,\n        )\n    )\n\n\ntop_by(\"n_donations\") | top_by(\"total_amount\") | top_by(\"mean_amount\") | top_by(\n    \"median_amount\"\n)\n```\n\n::: {.cell-output .cell-output-display execution_count=29}\n```{=html}\n\n<style>\n  #altair-viz-0d2671a035be4c5e8b9b39040d382bd3.vega-embed {\n    width: 100%;\n    display: flex;\n  }\n\n  #altair-viz-0d2671a035be4c5e8b9b39040d382bd3.vega-embed details,\n  #altair-viz-0d2671a035be4c5e8b9b39040d382bd3.vega-embed details summary {\n    position: relative;\n  }\n</style>\n<div id=\"altair-viz-0d2671a035be4c5e8b9b39040d382bd3\"></div>\n<script type=\"text/javascript\">\n  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n  (function(spec, embedOpt){\n    let outputDiv = document.currentScript.previousElementSibling;\n    if (outputDiv.id !== \"altair-viz-0d2671a035be4c5e8b9b39040d382bd3\") {\n      outputDiv = document.getElementById(\"altair-viz-0d2671a035be4c5e8b9b39040d382bd3\");\n    }\n    const paths = {\n      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.8.0?noext\",\n      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n    };\n\n    function maybeLoadScript(lib, version) {\n      var key = `${lib.replace(\"-\", \"\")}_version`;\n      return (VEGA_DEBUG[key] == version) ?\n        Promise.resolve(paths[lib]) :\n        new Promise(function(resolve, reject) {\n          var s = document.createElement('script');\n          document.getElementsByTagName(\"head\")[0].appendChild(s);\n          s.async = true;\n          s.onload = () => {\n            VEGA_DEBUG[key] = version;\n            return resolve(paths[lib]);\n          };\n          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n          s.src = paths[lib];\n        });\n    }\n\n    function showError(err) {\n      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n      throw err;\n    }\n\n    function displayChart(vegaEmbed) {\n      vegaEmbed(outputDiv, spec, embedOpt)\n        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n    }\n\n    if(typeof define === \"function\" && define.amd) {\n      requirejs.config({paths});\n      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n    } else {\n      maybeLoadScript(\"vega\", \"5\")\n        .then(() => maybeLoadScript(\"vega-lite\", \"5.8.0\"))\n        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n        .catch(showError)\n        .then(() => displayChart(vegaEmbed));\n    }\n  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"hconcat\": [{\"data\": {\"name\": \"data-88a0fd8958c48a49df7689732aa79f72\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"loc\", \"sort\": \"-y\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"n_donations\", \"type\": \"quantitative\"}}}, {\"data\": {\"name\": \"data-3702e7b9b7a3360d0314740591d3849d\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"loc\", \"sort\": \"-y\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"total_amount\", \"type\": \"quantitative\"}}}, {\"data\": {\"name\": \"data-290a9847dd1ffd50031c2439e64328e3\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"loc\", \"sort\": \"-y\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"mean_amount\", \"type\": \"quantitative\"}}}, {\"data\": {\"name\": \"data-5b715f99a3cfcd002ab9d2ec64e81179\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"loc\", \"sort\": \"-y\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"median_amount\", \"type\": \"quantitative\"}}}], \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.8.0.json\", \"datasets\": {\"data-88a0fd8958c48a49df7689732aa79f72\": [{\"loc\": \"NEW YORK, NY\", \"n_donations\": 695091, \"total_amount\": 444600108, \"mean_amount\": 639.6286356750411, \"median_amount\": 50}, {\"loc\": \"WASHINGTON, DC\", \"n_donations\": 401498, \"total_amount\": 124456508, \"mean_amount\": 309.9803934266173, \"median_amount\": 57}, {\"loc\": \"HOUSTON, TX\", \"n_donations\": 251960, \"total_amount\": 83026989, \"mean_amount\": 329.5244840450865, \"median_amount\": 50}, {\"loc\": \"LOS ANGELES, CA\", \"n_donations\": 245263, \"total_amount\": 89881980, \"mean_amount\": 366.4718282007478, \"median_amount\": 50}, {\"loc\": \"SAN FRANCISCO, CA\", \"n_donations\": 238117, \"total_amount\": 189799961, \"mean_amount\": 797.086982449804, \"median_amount\": 50}, {\"loc\": \"PHILADELPHIA, PA\", \"n_donations\": 222938, \"total_amount\": 36054977, \"mean_amount\": 161.72647552234253, \"median_amount\": 62}, {\"loc\": \"CHICAGO, IL\", \"n_donations\": 212527, \"total_amount\": 108119674, \"mean_amount\": 508.7338267608351, \"median_amount\": 40}, {\"loc\": \"SEATTLE, WA\", \"n_donations\": 197671, \"total_amount\": 52867387, \"mean_amount\": 267.4514066302088, \"median_amount\": 36}, {\"loc\": \"AUSTIN, TX\", \"n_donations\": 189865, \"total_amount\": 33315922, \"mean_amount\": 175.4716351091565, \"median_amount\": 38}, {\"loc\": \"ARLINGTON, VA\", \"n_donations\": 163168, \"total_amount\": 23382868, \"mean_amount\": 143.30547656403218, \"median_amount\": 50}], \"data-3702e7b9b7a3360d0314740591d3849d\": [{\"loc\": \"NEW YORK, NY\", \"n_donations\": 695091, \"total_amount\": 444600108, \"mean_amount\": 639.6286356750411, \"median_amount\": 50}, {\"loc\": \"SAN FRANCISCO, CA\", \"n_donations\": 238117, \"total_amount\": 189799961, \"mean_amount\": 797.086982449804, \"median_amount\": 50}, {\"loc\": \"LAS VEGAS, NV\", \"n_donations\": 65940, \"total_amount\": 153467387, \"mean_amount\": 2327.37923870185, \"median_amount\": 46}, {\"loc\": \"WASHINGTON, DC\", \"n_donations\": 401498, \"total_amount\": 124456508, \"mean_amount\": 309.9803934266173, \"median_amount\": 57}, {\"loc\": \"CHICAGO, IL\", \"n_donations\": 212527, \"total_amount\": 108119674, \"mean_amount\": 508.7338267608351, \"median_amount\": 40}, {\"loc\": \"LOS ANGELES, CA\", \"n_donations\": 245263, \"total_amount\": 89881980, \"mean_amount\": 366.4718282007478, \"median_amount\": 50}, {\"loc\": \"HOUSTON, TX\", \"n_donations\": 251960, \"total_amount\": 83026989, \"mean_amount\": 329.5244840450865, \"median_amount\": 50}, {\"loc\": \"DALLAS, TX\", \"n_donations\": 154038, \"total_amount\": 66558403, \"mean_amount\": 432.09080226956985, \"median_amount\": 57}, {\"loc\": \"SEATTLE, WA\", \"n_donations\": 197671, \"total_amount\": 52867387, \"mean_amount\": 267.4514066302088, \"median_amount\": 36}, {\"loc\": \"BOSTON, MA\", \"n_donations\": 82925, \"total_amount\": 47592049, \"mean_amount\": 573.9167802230932, \"median_amount\": 57}], \"data-290a9847dd1ffd50031c2439e64328e3\": [{\"loc\": \"LAKE FOREST, IL\", \"n_donations\": 5636, \"total_amount\": 37486362, \"mean_amount\": 6651.235273243435, \"median_amount\": 100}, {\"loc\": \"MOUNT VERNON, OH\", \"n_donations\": 1431, \"total_amount\": 5605857, \"mean_amount\": 3917.4402515723273, \"median_amount\": 47}, {\"loc\": \"LOS ALTOS HILLS, CA\", \"n_donations\": 4098, \"total_amount\": 10367629, \"mean_amount\": 2529.92410932162, \"median_amount\": 318}, {\"loc\": \"PALM BEACH, FL\", \"n_donations\": 7140, \"total_amount\": 17212425, \"mean_amount\": 2410.703781512605, \"median_amount\": 260}, {\"loc\": \"LAS VEGAS, NV\", \"n_donations\": 65940, \"total_amount\": 153467387, \"mean_amount\": 2327.37923870185, \"median_amount\": 45}, {\"loc\": \"RHINEBECK, NY\", \"n_donations\": 3014, \"total_amount\": 5942571, \"mean_amount\": 1971.6559389515594, \"median_amount\": 47}, {\"loc\": \"JOPLIN, MO\", \"n_donations\": 1839, \"total_amount\": 3617186, \"mean_amount\": 1966.9309407286569, \"median_amount\": 50}, {\"loc\": \"BALA CYNWYD, PA\", \"n_donations\": 3668, \"total_amount\": 6949933, \"mean_amount\": 1894.7472737186479, \"median_amount\": 100}, {\"loc\": \"CARMEL, IN\", \"n_donations\": 10932, \"total_amount\": 20383688, \"mean_amount\": 1864.5890962312478, \"median_amount\": 53}, {\"loc\": \"WAYLAND, MA\", \"n_donations\": 5283, \"total_amount\": 9704279, \"mean_amount\": 1836.8879424569373, \"median_amount\": 50}], \"data-5b715f99a3cfcd002ab9d2ec64e81179\": [{\"loc\": \"GLADWYNE, PA\", \"n_donations\": 1727, \"total_amount\": 1333243, \"mean_amount\": 771.9994209612044, \"median_amount\": 335}, {\"loc\": \"LOS ALTOS HILLS, CA\", \"n_donations\": 4098, \"total_amount\": 10367629, \"mean_amount\": 2529.92410932162, \"median_amount\": 326}, {\"loc\": \"MC LEAN, VA\", \"n_donations\": 4692, \"total_amount\": 3656109, \"mean_amount\": 779.2218670076726, \"median_amount\": 305}, {\"loc\": \"PALM BEACH, FL\", \"n_donations\": 7140, \"total_amount\": 17212425, \"mean_amount\": 2410.703781512605, \"median_amount\": 262}, {\"loc\": \"PARADISE VALLEY, AZ\", \"n_donations\": 8197, \"total_amount\": 7035291, \"mean_amount\": 858.2763206050994, \"median_amount\": 250}, {\"loc\": \"MISSION HILLS, KS\", \"n_donations\": 2258, \"total_amount\": 1642339, \"mean_amount\": 727.3423383525244, \"median_amount\": 250}, {\"loc\": \"DOVER, MA\", \"n_donations\": 1040, \"total_amount\": 976757, \"mean_amount\": 939.189423076923, \"median_amount\": 250}, {\"loc\": \"ATHERTON, CA\", \"n_donations\": 8780, \"total_amount\": 11595391, \"mean_amount\": 1320.6595671981777, \"median_amount\": 250}, {\"loc\": \"KENILWORTH, IL\", \"n_donations\": 1500, \"total_amount\": 855723, \"mean_amount\": 570.482, \"median_amount\": 250}, {\"loc\": \"WOODSIDE, CA\", \"n_donations\": 5626, \"total_amount\": 6110007, \"mean_amount\": 1086.030394596516, \"median_amount\": 249}]}}, {\"mode\": \"vega-lite\"});\n</script>\n```\n:::\n:::\n\n\n### By month\n\nWhen do the donations come in?\n\n::: {#6d0776d2 .cell execution_count=30}\n``` {.python .cell-code}\nby_month = summary_by(featured, _.date.month().name(\"month_int\"))\n# Sorta hacky, .substritute doesn't work to change dtypes (yet?)\n# so we cast to string and then do our mapping\nmonth_map = {\n    \"1\": \"Jan\",\n    \"2\": \"Feb\",\n    \"3\": \"Mar\",\n    \"4\": \"Apr\",\n    \"5\": \"May\",\n    \"6\": \"Jun\",\n    \"7\": \"Jul\",\n    \"8\": \"Aug\",\n    \"9\": \"Sep\",\n    \"10\": \"Oct\",\n    \"11\": \"Nov\",\n    \"12\": \"Dec\",\n}\nby_month = by_month.mutate(month_str=_.month_int.cast(str).substitute(month_map))\nby_month\n```\n\n::: {.cell-output .cell-output-display execution_count=30}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> month_int </span>┃<span style=\"font-weight: bold\"> n_donations </span>┃<span style=\"font-weight: bold\"> total_amount </span>┃<span style=\"font-weight: bold\"> mean_amount </span>┃<span style=\"font-weight: bold\"> median_amount </span>┃<span style=\"font-weight: bold\"> month_str </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int32</span>     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │\n├───────────┼─────────────┼──────────────┼─────────────┼───────────────┼───────────┤\n│      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1514</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">250297</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">165.321664</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">99</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>      │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">348979</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">174837854</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500.998209</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">122</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Jan      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">581646</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">255997655</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">440.126219</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Feb      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1042577</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">430906797</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">413.309326</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">81</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Mar      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1088244</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">299252692</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">274.986760</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Apr      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1374247</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">387317192</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">281.839576</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">48</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">May      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1667285</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">465305247</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">279.079610</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">44</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Jun      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1607053</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">320528605</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">199.451172</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Jul      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2023466</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">473544182</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">234.026261</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Aug      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2583847</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">697888624</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">270.096729</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">38</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Sep      </span> │\n│         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │\n└───────────┴─────────────┴──────────────┴─────────────┴───────────────┴───────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#a2b27c61 .cell execution_count=31}\n``` {.python .cell-code}\nmonths_in_order = list(month_map.values())\nalt.Chart(by_month.execute()).mark_bar().encode(\n    x=alt.X(\"month_str:O\", sort=months_in_order),\n    y=\"n_donations:Q\",\n)\n```\n\n::: {.cell-output .cell-output-display execution_count=31}\n```{=html}\n\n<style>\n  #altair-viz-8f3242609f02454fa698b7480db40ce3.vega-embed {\n    width: 100%;\n    display: flex;\n  }\n\n  #altair-viz-8f3242609f02454fa698b7480db40ce3.vega-embed details,\n  #altair-viz-8f3242609f02454fa698b7480db40ce3.vega-embed details summary {\n    position: relative;\n  }\n</style>\n<div id=\"altair-viz-8f3242609f02454fa698b7480db40ce3\"></div>\n<script type=\"text/javascript\">\n  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n  (function(spec, embedOpt){\n    let outputDiv = document.currentScript.previousElementSibling;\n    if (outputDiv.id !== \"altair-viz-8f3242609f02454fa698b7480db40ce3\") {\n      outputDiv = document.getElementById(\"altair-viz-8f3242609f02454fa698b7480db40ce3\");\n    }\n    const paths = {\n      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.8.0?noext\",\n      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n    };\n\n    function maybeLoadScript(lib, version) {\n      var key = `${lib.replace(\"-\", \"\")}_version`;\n      return (VEGA_DEBUG[key] == version) ?\n        Promise.resolve(paths[lib]) :\n        new Promise(function(resolve, reject) {\n          var s = document.createElement('script');\n          document.getElementsByTagName(\"head\")[0].appendChild(s);\n          s.async = true;\n          s.onload = () => {\n            VEGA_DEBUG[key] = version;\n            return resolve(paths[lib]);\n          };\n          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n          s.src = paths[lib];\n        });\n    }\n\n    function showError(err) {\n      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n      throw err;\n    }\n\n    function displayChart(vegaEmbed) {\n      vegaEmbed(outputDiv, spec, embedOpt)\n        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n    }\n\n    if(typeof define === \"function\" && define.amd) {\n      requirejs.config({paths});\n      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n    } else {\n      maybeLoadScript(\"vega\", \"5\")\n        .then(() => maybeLoadScript(\"vega-lite\", \"5.8.0\"))\n        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n        .catch(showError)\n        .then(() => displayChart(vegaEmbed));\n    }\n  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-60ff3524db1ce5308db1f55005eff874\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"month_str\", \"sort\": [\"Jan\", \"Feb\", \"Mar\", \"Apr\", \"May\", \"Jun\", \"Jul\", \"Aug\", \"Sep\", \"Oct\", \"Nov\", \"Dec\"], \"type\": \"ordinal\"}, \"y\": {\"field\": \"n_donations\", \"type\": \"quantitative\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.8.0.json\", \"datasets\": {\"data-60ff3524db1ce5308db1f55005eff874\": [{\"month_int\": null, \"n_donations\": 1514, \"total_amount\": 250297, \"mean_amount\": 165.3216644649934, \"median_amount\": 100, \"month_str\": null}, {\"month_int\": 1.0, \"n_donations\": 348979, \"total_amount\": 174837854, \"mean_amount\": 500.9982090612902, \"median_amount\": 120, \"month_str\": \"Jan\"}, {\"month_int\": 2.0, \"n_donations\": 581646, \"total_amount\": 255997655, \"mean_amount\": 440.126219384299, \"median_amount\": 100, \"month_str\": \"Feb\"}, {\"month_int\": 3.0, \"n_donations\": 1042577, \"total_amount\": 430906797, \"mean_amount\": 413.3093258339672, \"median_amount\": 81, \"month_str\": \"Mar\"}, {\"month_int\": 4.0, \"n_donations\": 1088244, \"total_amount\": 299252692, \"mean_amount\": 274.98676032213365, \"median_amount\": 50, \"month_str\": \"Apr\"}, {\"month_int\": 5.0, \"n_donations\": 1374247, \"total_amount\": 387317192, \"mean_amount\": 281.83957614606396, \"median_amount\": 48, \"month_str\": \"May\"}, {\"month_int\": 6.0, \"n_donations\": 1667285, \"total_amount\": 465305247, \"mean_amount\": 279.07960966481437, \"median_amount\": 44, \"month_str\": \"Jun\"}, {\"month_int\": 7.0, \"n_donations\": 1607053, \"total_amount\": 320528605, \"mean_amount\": 199.45117242555162, \"median_amount\": 35, \"month_str\": \"Jul\"}, {\"month_int\": 8.0, \"n_donations\": 2023466, \"total_amount\": 473544182, \"mean_amount\": 234.02626088108227, \"median_amount\": 35, \"month_str\": \"Aug\"}, {\"month_int\": 9.0, \"n_donations\": 2583847, \"total_amount\": 697888624, \"mean_amount\": 270.0967294116099, \"median_amount\": 38, \"month_str\": \"Sep\"}, {\"month_int\": 10.0, \"n_donations\": 3686024, \"total_amount\": 850820707, \"mean_amount\": 230.82343115508743, \"median_amount\": 29, \"month_str\": \"Oct\"}, {\"month_int\": 11.0, \"n_donations\": 2545616, \"total_amount\": 285143995, \"mean_amount\": 112.01375030640914, \"median_amount\": 25, \"month_str\": \"Nov\"}, {\"month_int\": 12.0, \"n_donations\": 2119311, \"total_amount\": 283081648, \"mean_amount\": 133.57249030463203, \"median_amount\": 25, \"month_str\": \"Dec\"}]}}, {\"mode\": \"vega-lite\"});\n</script>\n```\n:::\n:::\n\n\n## Conclusion\n\nThanks for following along! I hope you've learned something about Ibis, and\nmaybe even about campaign finance.\n\nIbis is a great tool for exploring data. I now find myself reaching for it\nwhen in the past I would have reached for pandas.\n\nSome of the highlights for me:\n\n- Fast, lazy execution, a great display format, and good type hinting/editor support for a great REPL experience.\n- Very well thought-out API and semantics (e.g. `isinstance(val, NumericValue)`?? That's beautiful!)\n- Fast and fairly complete string support, since I work with a lot of text data.\n- Extremely responsive maintainers. Sometimes I've submitted multiple feature requests and bug reports in a single day, and a PR has been merged by the next day.\n- Escape hatch to SQL. I didn't have to use that here, but if something isn't supported, you can always fall back to SQL.\n\nCheck out [The Ibis Website](https://ibis-project.org/) for more information.\n\n",
+    "engine": "jupyter",
+    "markdown": "---\ntitle: \"Exploring campaign finance data\"\nauthor: \"Nick Crews\"\ndate: \"2023-03-24\"\ncategories:\n    - blog\n    - data engineering\n    - case study\n    - duckdb\n    - performance\n---\n\nHi! My name is [Nick Crews](https://www.linkedin.com/in/nicholas-b-crews/),\nand I'm a data engineer that looks at public campaign finance data.\n\nIn this post, I'll walk through how I use Ibis to explore public campaign contribution\ndata from the Federal Election Commission (FEC). We'll do some loading,\ncleaning, featurizing, and visualization. There will be filtering, sorting, grouping,\nand aggregation.\n\n## Downloading The Data\n\n::: {#e29f35c8 .cell execution_count=2}\n``` {.python .cell-code}\nfrom pathlib import Path\nfrom zipfile import ZipFile\nfrom urllib.request import urlretrieve\n\n# Download and unzip the 2018 individual contributions data\nurl = \"https://cg-519a459a-0ea3-42c2-b7bc-fa1143481f74.s3-us-gov-west-1.amazonaws.com/bulk-downloads/2018/indiv18.zip\"\nzip_path = Path(\"indiv18.zip\")\ncsv_path = Path(\"indiv18.csv\")\n\nif not zip_path.exists():\n    urlretrieve(url, zip_path)\n\nif not csv_path.exists():\n    with ZipFile(zip_path) as zip_file, csv_path.open(\"w\") as csv_file:\n        for line in zip_file.open(\"itcont.txt\"):\n            csv_file.write(line.decode())\n```\n:::\n\n\n## Loading the data\n\nNow that we have our raw data in a .csv format, let's load it into Ibis,\nusing the duckdb backend.\n\nNote that a 4.3 GB .csv would be near the limit of what pandas could\nhandle on my laptop with 16GB of RAM. In pandas, typically every time\nyou perform a transformation on the data, a copy of the data is made.\nI could only do a few transformations before I ran out of memory.\n\nWith Ibis, this problem is solved in two different ways.\n\nFirst, because they are designed to work with very large datasets,\nmany (all?) SQL backends support out of core operations.\nThe data lives on disk, and are only loaded in a streaming fashion\nwhen needed, and then written back to disk as the operation is performed.\n\nSecond, unless you explicitly ask for it, Ibis makes use of lazy\nevaluation. This means that when you ask for a result, the\nresult is not persisted in memory. Only the original source\ndata is persisted. Everything else is derived from this on the fly.\n\n::: {#0a6991f4 .cell execution_count=3}\n``` {.python .cell-code}\nimport ibis\nfrom ibis import _\n\nibis.options.interactive = True\n\n# The raw .csv file doesn't have column names, so we will add them in the next step.\nraw = ibis.read_csv(csv_path)\nraw\n```\n\n::: {.cell-output .cell-output-display execution_count=16}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> C00401224 </span>┃<span style=\"font-weight: bold\"> A      </span>┃<span style=\"font-weight: bold\"> M6     </span>┃<span style=\"font-weight: bold\"> P      </span>┃<span style=\"font-weight: bold\"> 201804059101866001 </span>┃<span style=\"font-weight: bold\"> 24T    </span>┃<span style=\"font-weight: bold\"> IND    </span>┃<span style=\"font-weight: bold\"> STOUFFER, LEIGH   </span>┃<span style=\"font-weight: bold\"> AMSTELVEEN   </span>┃<span style=\"font-weight: bold\"> ZZ     </span>┃<span style=\"font-weight: bold\"> 1187RC    </span>┃<span style=\"font-weight: bold\"> MYSELF            </span>┃<span style=\"font-weight: bold\"> SELF EMPLOYED           </span>┃<span style=\"font-weight: bold\"> 05172017 </span>┃<span style=\"font-weight: bold\"> 10    </span>┃<span style=\"font-weight: bold\"> C00458000 </span>┃<span style=\"font-weight: bold\"> SA11AI_81445687 </span>┃<span style=\"font-weight: bold\"> 1217152 </span>┃<span style=\"font-weight: bold\"> column18 </span>┃<span style=\"font-weight: bold\"> EARMARKED FOR PROGRESSIVE CHANGE CAMPAIGN COMMITTEE (C00458000) </span>┃<span style=\"font-weight: bold\"> 4050820181544765358 </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>              │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                  │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>   │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>   │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>   │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                                          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>               │\n├───────────┼────────┼────────┼────────┼────────────────────┼────────┼────────┼───────────────────┼──────────────┼────────┼───────────┼───────────────────┼─────────────────────────┼──────────┼───────┼───────────┼─────────────────┼─────────┼──────────┼─────────────────────────────────────────────────────────────────┼─────────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101867748</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STRAWS, JOYCE    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">34761    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SILVERSEA CRUISES</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">RESERVATIONS SUPERVISOR</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05182017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81592336</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544770597</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101867748</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STRAWS, JOYCE    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">34761    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SILVERSEA CRUISES</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">RESERVATIONS SUPERVISOR</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81627562</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544770598</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865942</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">039020760</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE             </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE                   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05132017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81047921</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765179</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865942</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">039020760</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE             </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE                   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05152017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81209209</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765180</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865942</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">039020760</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE             </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE                   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81605223</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765181</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865943</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">039020760</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE             </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE                   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05242017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_82200022</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765182</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865943</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">03902    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED           </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05292017</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00213512</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_82589834</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR NANCY PELOSI FOR CONGRESS (C00213512)            </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765184</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101865944</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STOTT, JIM       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">039020760</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE             </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NONE                   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05302017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_82643727</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544765185</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101867050</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STRANGE, WINIFRED</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">34216    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED           </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05162017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81325918</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544768505</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">A     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">M6    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P     </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">201804059101867051</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">24T   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">STRANGE, WINIFRED</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">34216    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NOT EMPLOYED           </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05232017</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">C00000935</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">SA11AI_81991189</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1217152</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>     │ <span style=\"color: #008000; text-decoration-color: #008000\">EARMARKED FOR DCCC (C00000935)                                 </span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4050820181544768506</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                 │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                 │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>        │     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                                               │                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└───────────┴────────┴────────┴────────┴────────────────────┴────────┴────────┴───────────────────┴──────────────┴────────┴───────────┴───────────────────┴─────────────────────────┴──────────┴───────┴───────────┴─────────────────┴─────────┴──────────┴─────────────────────────────────────────────────────────────────┴─────────────────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#ebb6e702 .cell execution_count=4}\n``` {.python .cell-code}\n# For a more comprehesive description of the columns and their meaning, see\n# https://www.fec.gov/campaign-finance-data/contributions-individuals-file-description/\ncolumns = {\n    \"CMTE_ID\": \"keep\",  # Committee ID\n    \"AMNDT_IND\": \"drop\",  # Amendment indicator. A = amendment, N = new, T = termination\n    \"RPT_TP\": \"drop\",  # Report type (monthly, quarterly, etc)\n    \"TRANSACTION_PGI\": \"keep\",  # Primary/general indicator\n    \"IMAGE_NUM\": \"drop\",  # Image number\n    \"TRANSACTION_TP\": \"drop\",  # Transaction type\n    \"ENTITY_TP\": \"keep\",  # Entity type\n    \"NAME\": \"drop\",  # Contributor name\n    \"CITY\": \"keep\",  # Contributor city\n    \"STATE\": \"keep\",  # Contributor state\n    \"ZIP_CODE\": \"drop\",  # Contributor zip code\n    \"EMPLOYER\": \"drop\",  # Contributor employer\n    \"OCCUPATION\": \"drop\",  # Contributor occupation\n    \"TRANSACTION_DT\": \"keep\",  # Transaction date\n    \"TRANSACTION_AMT\": \"keep\",  # Transaction amount\n    # Other ID. For individual contributions will be null. For contributions from\n    # other FEC committees, will be the committee ID of the other committee.\n    \"OTHER_ID\": \"drop\",\n    \"TRAN_ID\": \"drop\",  # Transaction ID\n    \"FILE_NUM\": \"drop\",  # File number, unique number assigned to each report filed with the FEC\n    \"MEMO_CD\": \"drop\",  # Memo code\n    \"MEMO_TEXT\": \"drop\",  # Memo text\n    \"SUB_ID\": \"drop\",  # Submission ID. Unique number assigned to each transaction.\n}\n\nrenaming = {old: new for old, new in zip(raw.columns, columns.keys())}\nto_keep = [k for k, v in columns.items() if v == \"keep\"]\nkept = raw.relabel(renaming)[to_keep]\nkept\n```\n\n::: {.cell-output .cell-output-display execution_count=17}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CMTE_ID   </span>┃<span style=\"font-weight: bold\"> TRANSACTION_PGI </span>┃<span style=\"font-weight: bold\"> ENTITY_TP </span>┃<span style=\"font-weight: bold\"> CITY         </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_DT </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │\n├───────────┼─────────────────┼───────────┼──────────────┼────────┼────────────────┼─────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05182017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05132017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05152017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017      </span> │               <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05242017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05292017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05302017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05162017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05232017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>              │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└───────────┴─────────────────┴───────────┴──────────────┴────────┴────────────────┴─────────────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#3f4ad522 .cell execution_count=5}\n``` {.python .cell-code}\n# 21 million rows\nkept.count()\n```\n\n::: {.cell-output .cell-output-display}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=18}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre>┌──────────┐\n│ <span class=\"ansi-cyan-fg ansi-bold\">21730730</span> │\n└──────────┘</pre>\n```\n:::\n\n:::\n:::\n\n\nHuh, what's up with those timings? Previewing the head only took a fraction of a second,\nbut finding the number of rows took 10 seconds.\n\nThat's because duckdb is scanning the .csv file on the fly every time we access it.\nSo we only have to read the first few lines to get that preview,\nbut we have to read the whole file to get the number of rows.\n\nNote that this isn't a feature of Ibis, but a feature of Duckdb. This what I think is\none of the strengths of Ibis: Ibis itself doesn't have to implement any of the\noptimimizations or features of the backends. Those backends can focus on what they do\nbest, and Ibis can get those things for free.\n\nSo, let's tell duckdb to actually read in the file to its native format so later accesses\nwill be faster. This will be a ~20 seconds that we'll only have to pay once.\n\n::: {#c45e7319 .cell execution_count=6}\n``` {.python .cell-code}\nkept = kept.cache()\nkept\n```\n\n::: {.cell-output .cell-output-display execution_count=19}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CMTE_ID   </span>┃<span style=\"font-weight: bold\"> TRANSACTION_PGI </span>┃<span style=\"font-weight: bold\"> ENTITY_TP </span>┃<span style=\"font-weight: bold\"> CITY         </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_DT </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │\n├───────────┼─────────────────┼───────────┼──────────────┼────────┼────────────────┼─────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05182017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">OCOEE       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05132017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05152017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05192017      </span> │               <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05242017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05292017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CAPE NEDDICK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05302017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05162017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00401224</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ANNA MSRIA  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">05232017      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>              │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└───────────┴─────────────────┴───────────┴──────────────┴────────┴────────────────┴─────────────────┘\n</pre>\n```\n:::\n:::\n\n\nLook, now accessing it only takes a fraction of a second!\n\n::: {#881326dd .cell execution_count=7}\n``` {.python .cell-code}\nkept.count()\n```\n\n::: {.cell-output .cell-output-display}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=20}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre>┌──────────┐\n│ <span class=\"ansi-cyan-fg ansi-bold\">21730730</span> │\n└──────────┘</pre>\n```\n:::\n\n:::\n:::\n\n\n### Committees Data\n\nThe contributions only list an opaque `CMTE_ID` column. We want to know which actual\ncommittee this is. Let's load the committees table so we can lookup from\ncommittee ID to committee name.\n\n::: {#ae8760f6 .cell execution_count=8}\n``` {.python .cell-code}\ndef read_committees():\n    committees_url = \"https://cg-519a459a-0ea3-42c2-b7bc-fa1143481f74.s3-us-gov-west-1.amazonaws.com/bulk-downloads/2018/committee_summary_2018.csv\"\n    # This just creates a view, it doesn't actually fetch the data yet\n    tmp = ibis.read_csv(committees_url)\n    tmp = tmp[\"CMTE_ID\", \"CMTE_NM\"]\n    # The raw table contains multiple rows for each committee id, so lets pick\n    # an arbitrary row for each committee id as the representative name.\n    deduped = tmp.group_by(\"CMTE_ID\").agg(CMTE_NM=_.CMTE_NM.arbitrary())\n    return deduped\n\n\ncomms = read_committees().cache()\ncomms\n```\n\n::: {.cell-output .cell-output-display execution_count=21}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CMTE_ID   </span>┃<span style=\"font-weight: bold\"> CMTE_NM                                                        </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                                         │\n├───────────┼────────────────────────────────────────────────────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00659441</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">JASON ORTITAY FOR CONGRESS                                    </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00297911</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">TEXAS FORESTRY ASSOCIATION FORESTRY POLITICAL ACTION COMMITTEE</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00340745</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">WADDELL &amp; REED FINANCIAL, INC. POLITICAL ACTION COMMITTEE     </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00679217</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CANTWELL-WARREN VICTORY FUND                                  </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00101204</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NATIONAL FISHERIES INSTITUTE (FISHPAC)                        </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00010520</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">MEREDITH CORPORATION EMPLOYEES FUND FOR BETTER GOVERNMENT     </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00532788</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">LAFAYETTE COUNTY DEMOCRATIC PARTY                             </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00128561</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">TOLL BROS. INC. PAC                                           </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00510958</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">WENDYROGERS.ORG                                               </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">C00665604</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">COMMITTEE TO ELECT BILL EBBEN                                 </span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                                              │\n└───────────┴────────────────────────────────────────────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\nNow add the committee name to the contributions table:\n\n::: {#8fe204d4 .cell execution_count=9}\n``` {.python .cell-code}\ntogether = kept.left_join(comms, \"CMTE_ID\").drop(\"CMTE_ID\", \"CMTE_ID_right\")\ntogether\n```\n\n::: {.cell-output .cell-output-display execution_count=22}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> TRANSACTION_PGI </span>┃<span style=\"font-weight: bold\"> ENTITY_TP </span>┃<span style=\"font-weight: bold\"> CITY             </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_DT </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃<span style=\"font-weight: bold\"> CMTE_NM                                         </span>┃\n┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                          │\n├─────────────────┼───────────┼──────────────────┼────────┼────────────────┼─────────────────┼─────────────────────────────────────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">COHASSET        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">MA    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">KEY LARGO       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01042017      </span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5000</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">LOOKOUT MOUNTAIN</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NORTH YARMOUTH  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ALPHARETTA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">HOLLIS CENTER   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ALEXANDRIA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">VA    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">01312017      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>              │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                               │\n└─────────────────┴───────────┴──────────────────┴────────┴────────────────┴─────────────────┴─────────────────────────────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\n## Cleaning\n\nFirst, let's drop any contributions that don't have a committee name. There are only 6 of them.\n\n::: {#215670b2 .cell execution_count=10}\n``` {.python .cell-code}\n# We can do this fearlessly, no .copy() needed, because\n# everything in Ibis is immutable. If we did this in pandas,\n# we might start modifying the original DataFrame accidentally!\ncleaned = together\n\nhas_name = cleaned.CMTE_NM.notnull()\ncleaned = cleaned[has_name]\nhas_name.value_counts()\n```\n\n::: {.cell-output .cell-output-display execution_count=23}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> NotNull(CMTE_NM) </span>┃<span style=\"font-weight: bold\"> NotNull(CMTE_NM)_count </span>┃\n┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">boolean</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>                  │\n├──────────────────┼────────────────────────┤\n│ True             │               <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21730724</span> │\n│ False            │                      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span> │\n└──────────────────┴────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\nLet's look at the `ENTITY_TP` column. This represents the type of entity that\nmade the contribution:\n\n::: {#8e39507b .cell execution_count=11}\n``` {.python .cell-code}\ntogether.ENTITY_TP.value_counts()\n```\n\n::: {.cell-output .cell-output-display execution_count=24}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> ENTITY_TP </span>┃<span style=\"font-weight: bold\"> ENTITY_TP_count </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │\n├───────────┼─────────────────┤\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>      │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5289</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">CAN      </span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13659</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">COM      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">867</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">IND      </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21687992</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">ORG      </span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18555</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">PAC      </span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3621</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">PTY      </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">49</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">CCM      </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">698</span> │\n└───────────┴─────────────────┘\n</pre>\n```\n:::\n:::\n\n\nWe only care about contributions from individuals.\n\nOnce we filter on this column, the contents of it are irrelevant, so let's drop it.\n\n::: {#e1453e27 .cell execution_count=12}\n``` {.python .cell-code}\ncleaned = together[_.ENTITY_TP == \"IND\"].drop(\"ENTITY_TP\")\n```\n:::\n\n\nIt looks like the `TRANSACTION_DT` column was a raw string like \"MMDDYYYY\",\nso let's convert that to a proper date type.\n\n::: {#bf3dadc7 .cell execution_count=13}\n``` {.python .cell-code}\nfrom ibis.expr.types import StringValue, DateValue\n\n\ndef mmddyyyy_to_date(val: StringValue) -> DateValue:\n    return val.cast(str).lpad(8, \"0\").to_timestamp(\"%m%d%Y\").date()\n\n\ncleaned = cleaned.mutate(date=mmddyyyy_to_date(_.TRANSACTION_DT)).drop(\"TRANSACTION_DT\")\ncleaned\n```\n\n::: {.cell-output .cell-output-display execution_count=26}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> TRANSACTION_PGI </span>┃<span style=\"font-weight: bold\"> CITY             </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃<span style=\"font-weight: bold\"> CMTE_NM                                         </span>┃<span style=\"font-weight: bold\"> date       </span>┃\n┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">date</span>       │\n├─────────────────┼──────────────────┼────────┼─────────────────┼─────────────────────────────────────────────────┼────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">COHASSET        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">MA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">KEY LARGO       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FL    </span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5000</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-04</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">LOOKOUT MOUNTAIN</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NORTH YARMOUTH  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ALPHARETTA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">HOLLIS CENTER   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">FALMOUTH        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ME    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">ALEXANDRIA      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">VA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">UNUM GROUP POLITICAL ACTION COMMITTEE (UNUMPAC)</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-01-31</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                               │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>          │\n└─────────────────┴──────────────────┴────────┴─────────────────┴─────────────────────────────────────────────────┴────────────┘\n</pre>\n```\n:::\n:::\n\n\nThe `TRANSACTION_PGI` column represents the type (primary, general, etc) of election,\nand the year. But it seems to be not very consistent:\n\n::: {#6cb98e2b .cell execution_count=14}\n``` {.python .cell-code}\ncleaned.TRANSACTION_PGI.topk(10)\n```\n\n::: {.cell-output .cell-output-display execution_count=27}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> TRANSACTION_PGI </span>┃<span style=\"font-weight: bold\"> CountStar() </span>┃\n┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │\n├─────────────────┼─────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P              </span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">17013596</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">G2018          </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2095123</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P2018          </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1677183</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P2020          </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">208501</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">O2018          </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">161874</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">S2017          </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">124336</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">G2017          </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">98401</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P2022          </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">91136</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">P2017          </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">61153</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">R2017          </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">54281</span> │\n└─────────────────┴─────────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#463caa6b .cell execution_count=15}\n``` {.python .cell-code}\ndef get_election_type(pgi: StringValue) -> StringValue:\n    \"\"\"Use the first letter of the TRANSACTION_PGI column to determine the election type\n\n    If the first letter is not one of the known election stage, then return null.\n    \"\"\"\n    election_types = {\n        \"P\": \"primary\",\n        \"G\": \"general\",\n        \"O\": \"other\",\n        \"C\": \"convention\",\n        \"R\": \"runoff\",\n        \"S\": \"special\",\n        \"E\": \"recount\",\n    }\n    first_letter = pgi[0]\n    return first_letter.substitute(election_types, else_=ibis.null())\n\n\ncleaned = cleaned.mutate(election_type=get_election_type(_.TRANSACTION_PGI)).drop(\n    \"TRANSACTION_PGI\"\n)\ncleaned\n```\n\n::: {.cell-output .cell-output-display execution_count=28}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CITY       </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃<span style=\"font-weight: bold\"> CMTE_NM                   </span>┃<span style=\"font-weight: bold\"> date       </span>┃<span style=\"font-weight: bold\"> election_type </span>┃\n┡━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                    │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">date</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │\n├────────────┼────────┼─────────────────┼───────────────────────────┼────────────┼───────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">ATLANTA   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">GA    </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NANCY PELOSI FOR CONGRESS</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-20</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">AUSTIN    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">TX    </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NANCY PELOSI FOR CONGRESS</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-04</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">WASHINGTON</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">DC    </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NANCY PELOSI FOR CONGRESS</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-23</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">HONOLULU  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">HI    </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NANCY PELOSI FOR CONGRESS</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-04-20</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">MAMARONECK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NY    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NANCY PELOSI FOR CONGRESS</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-02</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">REHOBOTH  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">MA    </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NANCY PELOSI FOR CONGRESS</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-01</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">BERKELEY  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CA    </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NANCY PELOSI FOR CONGRESS</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-05</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">BEAUMONT  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">TX    </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NANCY PELOSI FOR CONGRESS</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-04-12</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">CONCORD   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">MA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">200</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NANCY PELOSI FOR CONGRESS</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-05-05</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">OXNARD    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CA    </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NANCY PELOSI FOR CONGRESS</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-03-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │\n└────────────┴────────┴─────────────────┴───────────────────────────┴────────────┴───────────────┘\n</pre>\n```\n:::\n:::\n\n\nThat worked well! There are 0 nulls in the resulting column, so we always were\nable to determine the election type.\n\n::: {#ead49c9e .cell execution_count=16}\n``` {.python .cell-code}\ncleaned.election_type.topk(10)\n```\n\n::: {.cell-output .cell-output-display execution_count=29}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> election_type </span>┃<span style=\"font-weight: bold\"> CountStar() </span>┃\n┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │\n├───────────────┼─────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">19061953</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2216685</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">other        </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">161965</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">special      </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">149572</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">runoff       </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">69637</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">convention   </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">22453</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">recount      </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5063</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>          │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">664</span> │\n└───────────────┴─────────────┘\n</pre>\n```\n:::\n:::\n\n\nAbout 1/20 of transactions are negative. These could represent refunds, or they\ncould be data entry errors. Let's drop them to keep it simple.\n\n::: {#ee56a3f3 .cell execution_count=17}\n``` {.python .cell-code}\nabove_zero = cleaned.TRANSACTION_AMT > 0\ncleaned = cleaned[above_zero]\nabove_zero.value_counts()\n```\n\n::: {.cell-output .cell-output-display execution_count=30}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> Greater(TRANSACTION_AMT, 0) </span>┃<span style=\"font-weight: bold\"> Greater(TRANSACTION_AMT, 0)_count </span>┃\n┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">boolean</span>                     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>                             │\n├─────────────────────────────┼───────────────────────────────────┤\n│ True                        │                          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20669809</span> │\n│ False                       │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1018183</span> │\n└─────────────────────────────┴───────────────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\n## Adding Features\n\nNow that the data is cleaned up to a usable format, let's add some features.\n\nFirst, it's useful to categorize donations by size, placing them into buckets\nof small, medium, large, etc.\n\n::: {#0ccc57df .cell execution_count=18}\n``` {.python .cell-code}\nedges = [\n    10,\n    50,\n    100,\n    500,\n    1000,\n    5000,\n]\nlabels = [\n    \"<10\",\n    \"10-50\",\n    \"50-100\",\n    \"100-500\",\n    \"500-1000\",\n    \"1000-5000\",\n    \"5000+\",\n]\n\n\ndef bucketize(vals, edges, str_labels):\n    # Uses Ibis's .bucket() method to create a categorical column\n    int_labels = vals.bucket(edges, include_under=True, include_over=True)\n    # Map the integer labels to the string labels\n    int_to_str = {str(i): s for i, s in enumerate(str_labels)}\n    return int_labels.cast(str).substitute(int_to_str)\n\n\nfeatured = cleaned.mutate(amount_bucket=bucketize(_.TRANSACTION_AMT, edges, labels))\nfeatured\n```\n\n::: {.cell-output .cell-output-display execution_count=31}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CITY         </span>┃<span style=\"font-weight: bold\"> STATE  </span>┃<span style=\"font-weight: bold\"> TRANSACTION_AMT </span>┃<span style=\"font-weight: bold\"> CMTE_NM               </span>┃<span style=\"font-weight: bold\"> date       </span>┃<span style=\"font-weight: bold\"> election_type </span>┃<span style=\"font-weight: bold\"> amount_bucket </span>┃\n┡━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">date</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │\n├──────────────┼────────┼─────────────────┼───────────────────────┼────────────┼───────────────┼───────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">REMINGTON   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IN    </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">AMERICA'S LIBERTY PAC</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-05-30</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">50-100       </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">REMINGTON   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">IN    </span> │              <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">AMERICA'S LIBERTY PAC</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-05</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">50-100       </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">VANCOUVER   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">WA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">AMERICA'S LIBERTY PAC</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-07</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">SOLANA BEACH</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">CA    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">AMERICA'S LIBERTY PAC</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-26</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">HILLSDALE   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">MI    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">250</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">AMERICA'S LIBERTY PAC</span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-05-15</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">MIDDLEBURY  </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">VT    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NBT PAC FEDERAL FUND </span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-05</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">WILLISTON   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">VT    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NBT PAC FEDERAL FUND </span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-05-30</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">GLENMONT    </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NY    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">350</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NBT PAC FEDERAL FUND </span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-01</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">NORWICH     </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NY    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">250</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NBT PAC FEDERAL FUND </span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-05-31</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">CLIFTON PARK</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NY    </span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">250</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">NBT PAC FEDERAL FUND </span> │ <span style=\"color: #800080; text-decoration-color: #800080\">2017-06-26</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>            │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>      │               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │\n└──────────────┴────────┴─────────────────┴───────────────────────┴────────────┴───────────────┴───────────────┘\n</pre>\n```\n:::\n:::\n\n\n## Analysis\n\n### By donation size\n\nOne thing we can look at is the donation breakdown by size:\n- Are most donations small or large?\n- Where do politicians/committees get most of their money from? Large or small donations?\n\nWe also will compare performance of Ibis vs pandas during this groupby.\n\n::: {#6c9dae32 .cell execution_count=19}\n``` {.python .cell-code}\ndef summary_by(table, by):\n    return table.group_by(by).agg(\n        n_donations=_.count(),\n        total_amount=_.TRANSACTION_AMT.sum(),\n        mean_amount=_.TRANSACTION_AMT.mean(),\n        median_amount=_.TRANSACTION_AMT.approx_median(),\n    )\n\n\ndef summary_by_pandas(df, by):\n    return df.groupby(by, as_index=False).agg(\n        n_donations=(\"election_type\", \"count\"),\n        total_amount=(\"TRANSACTION_AMT\", \"sum\"),\n        mean_amount=(\"TRANSACTION_AMT\", \"mean\"),\n        median_amount=(\"TRANSACTION_AMT\", \"median\"),\n    )\n\n\n# persist the input data so the following timings of the group_by are accurate.\nsubset = featured[\"election_type\", \"amount_bucket\", \"TRANSACTION_AMT\"]\nsubset = subset.cache()\npandas_subset = subset.execute()\n```\n:::\n\n\nLet's take a look at what we are actually computing:\n\n::: {#1b310e3e .cell execution_count=20}\n``` {.python .cell-code}\nby_type_and_bucket = summary_by(subset, [\"election_type\", \"amount_bucket\"])\nby_type_and_bucket\n```\n\n::: {.cell-output .cell-output-display execution_count=33}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> election_type </span>┃<span style=\"font-weight: bold\"> amount_bucket </span>┃<span style=\"font-weight: bold\"> n_donations </span>┃<span style=\"font-weight: bold\"> total_amount </span>┃<span style=\"font-weight: bold\"> mean_amount  </span>┃<span style=\"font-weight: bold\"> median_amount </span>┃\n┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>         │\n├───────────────┼───────────────┼─────────────┼──────────────┼──────────────┼───────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">634677</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">334630687</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">527.245649</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">5000+        </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3125</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">44496373</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14238.839360</span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7537</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">special      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7811</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4003293</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512.519908</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">runoff       </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18193</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3088289</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">169.751498</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">convention   </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1824</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">945321</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">518.268092</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">&lt;10          </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115873</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">536742</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.632158</span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">50-100       </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">304363</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">16184312</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">53.174374</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">1000-5000    </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">246101</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">460025242</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1869.253851</span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1978</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">10-50        </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">660787</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14411588</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21.809733</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">other        </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">119</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">62535</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">525.504202</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└───────────────┴───────────────┴─────────────┴──────────────┴──────────────┴───────────────┘\n</pre>\n```\n:::\n:::\n\n\nOK, now let's do our timings.\n\nOne interesting thing to pay attention to here is the execution time for the following\ngroupby. Before, we could get away with lazy execution: because we only wanted to preview\nthe first few rows, we only had to compute the first few rows, so all our previews were\nvery fast.\n\nBut now, as soon as we do a groupby, we have to actually go through the whole dataset\nin order to compute the aggregate per group. So this is going to be slower. BUT,\nduckdb is still quite fast. It only takes milliseconds to groupby-agg all 20 million rows!\n\n::: {#32424707 .cell execution_count=21}\n``` {.python .cell-code}\n%timeit summary_by(subset, [\"election_type\", \"amount_bucket\"]).execute()  # .execute() so we actually fetch the data\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n161 ms ± 4.75 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n```\n:::\n:::\n\n\nNow let's try the same thing in pandas:\n\n::: {#cc653b7f .cell execution_count=22}\n``` {.python .cell-code}\n%timeit summary_by_pandas(pandas_subset, [\"election_type\", \"amount_bucket\"])\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n2.19 s ± 6.54 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n```\n:::\n:::\n\n\nIt takes about 4 seconds, which is about 10 times slower than duckdb.\nAt this scale, it again doesn't matter,\nbut you could imagine with a dataset much larger than this, it would matter.\n\nLet's also think about memory usage:\n\n::: {#c967896c .cell execution_count=23}\n``` {.python .cell-code}\npandas_subset.memory_usage(deep=True).sum() / 1e9  # GB\n```\n\n::: {.cell-output .cell-output-display execution_count=36}\n```\n2.782586667\n```\n:::\n:::\n\n\nThe source dataframe is couple gigabytes, so probably during the groupby,\nthe peak memory usage is going to be a bit higher than this. You could use a profiler\nsuch as [FIL](https://github.com/pythonspeed/filprofiler) if you wanted an exact number,\nI was too lazy to use that here.\n\nAgain, this works on my laptop at this dataset size, but much larger than this and I'd\nstart having problems. Duckdb on the other hand is designed around working out of core\nso it should scale to datasets into the hundreds of gigabytes, much larger than your\ncomputer's RAM.\n\n### Back to analysis\n\nOK, let's plot the result of that groupby.\n\nSurprise! (Or maybe not...) Most donations are small. But most of the money comes\nfrom donations larger than $1000.\n\nWell if that's the case, why do politicians spend so much time soliciting small\ndonations? One explanation is that they can use the number of donations\nas a marketing pitch, to show how popular they are, and thus how viable of a\ncandidate they are.\n\nThis also might explain whose interests are being served by our politicians.\n\n::: {#6808107a .cell execution_count=24}\n``` {.python .cell-code}\nimport altair as alt\n\n# Do some bookkeeping so the buckets are displayed smallest to largest on the charts\nbucket_col = alt.Column(\"amount_bucket:N\", sort=labels)\n\nn_by_bucket = (\n    alt.Chart(by_type_and_bucket.execute())\n    .mark_bar()\n    .encode(\n        x=bucket_col,\n        y=\"n_donations:Q\",\n        color=\"election_type:N\",\n    )\n)\ntotal_by_bucket = (\n    alt.Chart(by_type_and_bucket.execute())\n    .mark_bar()\n    .encode(\n        x=bucket_col,\n        y=\"total_amount:Q\",\n        color=\"election_type:N\",\n    )\n)\nn_by_bucket | total_by_bucket\n```\n\n::: {.cell-output .cell-output-display execution_count=37}\n```{=html}\n\n<style>\n  #altair-viz-6fbf17b0e95f4f8c9babe5bb35792a50.vega-embed {\n    width: 100%;\n    display: flex;\n  }\n\n  #altair-viz-6fbf17b0e95f4f8c9babe5bb35792a50.vega-embed details,\n  #altair-viz-6fbf17b0e95f4f8c9babe5bb35792a50.vega-embed details summary {\n    position: relative;\n  }\n</style>\n<div id=\"altair-viz-6fbf17b0e95f4f8c9babe5bb35792a50\"></div>\n<script type=\"text/javascript\">\n  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n  (function(spec, embedOpt){\n    let outputDiv = document.currentScript.previousElementSibling;\n    if (outputDiv.id !== \"altair-viz-6fbf17b0e95f4f8c9babe5bb35792a50\") {\n      outputDiv = document.getElementById(\"altair-viz-6fbf17b0e95f4f8c9babe5bb35792a50\");\n    }\n    const paths = {\n      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.16.3?noext\",\n      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n    };\n\n    function maybeLoadScript(lib, version) {\n      var key = `${lib.replace(\"-\", \"\")}_version`;\n      return (VEGA_DEBUG[key] == version) ?\n        Promise.resolve(paths[lib]) :\n        new Promise(function(resolve, reject) {\n          var s = document.createElement('script');\n          document.getElementsByTagName(\"head\")[0].appendChild(s);\n          s.async = true;\n          s.onload = () => {\n            VEGA_DEBUG[key] = version;\n            return resolve(paths[lib]);\n          };\n          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n          s.src = paths[lib];\n        });\n    }\n\n    function showError(err) {\n      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n      throw err;\n    }\n\n    function displayChart(vegaEmbed) {\n      vegaEmbed(outputDiv, spec, embedOpt)\n        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n    }\n\n    if(typeof define === \"function\" && define.amd) {\n      requirejs.config({paths});\n      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n    } else {\n      maybeLoadScript(\"vega\", \"5\")\n        .then(() => maybeLoadScript(\"vega-lite\", \"5.16.3\"))\n        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n        .catch(showError)\n        .then(() => displayChart(vegaEmbed));\n    }\n  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"hconcat\": [{\"data\": {\"name\": \"data-ec402682d040f07539df5cc760e76274\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"color\": {\"field\": \"election_type\", \"type\": \"nominal\"}, \"x\": {\"field\": \"amount_bucket\", \"sort\": [\"<10\", \"10-50\", \"50-100\", \"100-500\", \"500-1000\", \"1000-5000\", \"5000+\"], \"type\": \"nominal\"}, \"y\": {\"field\": \"n_donations\", \"type\": \"quantitative\"}}}, {\"data\": {\"name\": \"data-c12eefc8ce67300e6225801f6dacde98\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"color\": {\"field\": \"election_type\", \"type\": \"nominal\"}, \"x\": {\"field\": \"amount_bucket\", \"sort\": [\"<10\", \"10-50\", \"50-100\", \"100-500\", \"500-1000\", \"1000-5000\", \"5000+\"], \"type\": \"nominal\"}, \"y\": {\"field\": \"total_amount\", \"type\": \"quantitative\"}}}], \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.16.3.json\", \"datasets\": {\"data-ec402682d040f07539df5cc760e76274\": [{\"election_type\": \"general\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 246101, \"total_amount\": 460025242, \"mean_amount\": 1869.2538510611496, \"median_amount\": 1965}, {\"election_type\": \"general\", \"amount_bucket\": \"10-50\", \"n_donations\": 660787, \"total_amount\": 14411588, \"mean_amount\": 21.809732939661345, \"median_amount\": 25}, {\"election_type\": \"general\", \"amount_bucket\": \"50-100\", \"n_donations\": 304363, \"total_amount\": 16184312, \"mean_amount\": 53.174374020495264, \"median_amount\": 50}, {\"election_type\": \"general\", \"amount_bucket\": \"<10\", \"n_donations\": 115873, \"total_amount\": 536742, \"mean_amount\": 4.632157620843509, \"median_amount\": 5}, {\"election_type\": \"runoff\", \"amount_bucket\": \"5000+\", \"n_donations\": 37, \"total_amount\": 211400, \"mean_amount\": 5713.513513513513, \"median_amount\": 5400}, {\"election_type\": \"other\", \"amount_bucket\": \"500-1000\", \"n_donations\": 119, \"total_amount\": 62535, \"mean_amount\": 525.5042016806723, \"median_amount\": 500}, {\"election_type\": \"special\", \"amount_bucket\": \"500-1000\", \"n_donations\": 7811, \"total_amount\": 4003293, \"mean_amount\": 512.5199078223019, \"median_amount\": 500}, {\"election_type\": \"convention\", \"amount_bucket\": \"500-1000\", \"n_donations\": 1824, \"total_amount\": 945321, \"mean_amount\": 518.2680921052631, \"median_amount\": 500}, {\"election_type\": \"runoff\", \"amount_bucket\": \"100-500\", \"n_donations\": 18193, \"total_amount\": 3088289, \"mean_amount\": 169.75149782883526, \"median_amount\": 101}, {\"election_type\": \"general\", \"amount_bucket\": \"100-500\", \"n_donations\": 700821, \"total_amount\": 123174568, \"mean_amount\": 175.75753009684357, \"median_amount\": 149}, {\"election_type\": null, \"amount_bucket\": \"500-1000\", \"n_donations\": 89, \"total_amount\": 48290, \"mean_amount\": 542.5842696629213, \"median_amount\": 500}, {\"election_type\": \"general\", \"amount_bucket\": \"500-1000\", \"n_donations\": 174182, \"total_amount\": 91015697, \"mean_amount\": 522.5321617618354, \"median_amount\": 500}, {\"election_type\": \"primary\", \"amount_bucket\": \"5000+\", \"n_donations\": 44085, \"total_amount\": 1558371116, \"mean_amount\": 35349.237064761255, \"median_amount\": 10000}, {\"election_type\": \"recount\", \"amount_bucket\": \"5000+\", \"n_donations\": 26, \"total_amount\": 1888024, \"mean_amount\": 72616.30769230769, \"median_amount\": 101450}, {\"election_type\": null, \"amount_bucket\": \"100-500\", \"n_donations\": 195, \"total_amount\": 46746, \"mean_amount\": 239.72307692307692, \"median_amount\": 250}, {\"election_type\": \"other\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 235, \"total_amount\": 548212, \"mean_amount\": 2332.817021276596, \"median_amount\": 2633}, {\"election_type\": \"other\", \"amount_bucket\": \"50-100\", \"n_donations\": 451, \"total_amount\": 27149, \"mean_amount\": 60.19733924611973, \"median_amount\": 50}, {\"election_type\": \"other\", \"amount_bucket\": \"10-50\", \"n_donations\": 2644, \"total_amount\": 64297, \"mean_amount\": 24.318078668683814, \"median_amount\": 23}, {\"election_type\": \"other\", \"amount_bucket\": \"<10\", \"n_donations\": 10993, \"total_amount\": 25816, \"mean_amount\": 2.3484035295187846, \"median_amount\": 1}, {\"election_type\": \"special\", \"amount_bucket\": \"5000+\", \"n_donations\": 129, \"total_amount\": 788712, \"mean_amount\": 6114.046511627907, \"median_amount\": 5400}, {\"election_type\": null, \"amount_bucket\": \"1000-5000\", \"n_donations\": 116, \"total_amount\": 228657, \"mean_amount\": 1971.1810344827586, \"median_amount\": 1300}, {\"election_type\": \"convention\", \"amount_bucket\": \"5000+\", \"n_donations\": 219, \"total_amount\": 1590300, \"mean_amount\": 7261.643835616438, \"median_amount\": 8100}, {\"election_type\": \"other\", \"amount_bucket\": \"100-500\", \"n_donations\": 630, \"total_amount\": 117988, \"mean_amount\": 187.2825396825397, \"median_amount\": 192}, {\"election_type\": null, \"amount_bucket\": \"<10\", \"n_donations\": 24, \"total_amount\": 108, \"mean_amount\": 4.5, \"median_amount\": 5}, {\"election_type\": null, \"amount_bucket\": \"10-50\", \"n_donations\": 151, \"total_amount\": 3167, \"mean_amount\": 20.973509933774835, \"median_amount\": 25}, {\"election_type\": null, \"amount_bucket\": \"50-100\", \"n_donations\": 36, \"total_amount\": 1880, \"mean_amount\": 52.22222222222222, \"median_amount\": 50}, {\"election_type\": \"primary\", \"amount_bucket\": \"500-1000\", \"n_donations\": 634677, \"total_amount\": 334630687, \"mean_amount\": 527.2456493618014, \"median_amount\": 500}, {\"election_type\": \"runoff\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 5196, \"total_amount\": 9601993, \"mean_amount\": 1847.958622016936, \"median_amount\": 1913}, {\"election_type\": \"runoff\", \"amount_bucket\": \"10-50\", \"n_donations\": 20166, \"total_amount\": 461107, \"mean_amount\": 22.865565803828225, \"median_amount\": 25}, {\"election_type\": \"general\", \"amount_bucket\": \"5000+\", \"n_donations\": 3125, \"total_amount\": 44496373, \"mean_amount\": 14238.83936, \"median_amount\": 7534}, {\"election_type\": \"runoff\", \"amount_bucket\": \"<10\", \"n_donations\": 10191, \"total_amount\": 49621, \"mean_amount\": 4.869100186439015, \"median_amount\": 5}, {\"election_type\": \"runoff\", \"amount_bucket\": \"50-100\", \"n_donations\": 11578, \"total_amount\": 585827, \"mean_amount\": 50.59828986007946, \"median_amount\": 50}, {\"election_type\": \"recount\", \"amount_bucket\": \"500-1000\", \"n_donations\": 494, \"total_amount\": 250960, \"mean_amount\": 508.0161943319838, \"median_amount\": 500}, {\"election_type\": \"primary\", \"amount_bucket\": \"100-500\", \"n_donations\": 3636287, \"total_amount\": 637353634, \"mean_amount\": 175.27594329050484, \"median_amount\": 149}, {\"election_type\": \"convention\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 2822, \"total_amount\": 4977314, \"mean_amount\": 1763.7540751240256, \"median_amount\": 1440}, {\"election_type\": \"special\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 7935, \"total_amount\": 13493154, \"mean_amount\": 1700.4604914933836, \"median_amount\": 1000}, {\"election_type\": \"convention\", \"amount_bucket\": \"50-100\", \"n_donations\": 2966, \"total_amount\": 153281, \"mean_amount\": 51.67936614969656, \"median_amount\": 50}, {\"election_type\": \"special\", \"amount_bucket\": \"10-50\", \"n_donations\": 51066, \"total_amount\": 1134616, \"mean_amount\": 22.21861904202405, \"median_amount\": 25}, {\"election_type\": \"special\", \"amount_bucket\": \"<10\", \"n_donations\": 25115, \"total_amount\": 122898, \"mean_amount\": 4.893410312562214, \"median_amount\": 5}, {\"election_type\": \"special\", \"amount_bucket\": \"50-100\", \"n_donations\": 22859, \"total_amount\": 1177660, \"mean_amount\": 51.518439126820944, \"median_amount\": 50}, {\"election_type\": null, \"amount_bucket\": \"5000+\", \"n_donations\": 48, \"total_amount\": 1622455, \"mean_amount\": 33801.145833333336, \"median_amount\": 21731}, {\"election_type\": \"convention\", \"amount_bucket\": \"10-50\", \"n_donations\": 6848, \"total_amount\": 141604, \"mean_amount\": 20.678154205607477, \"median_amount\": 25}, {\"election_type\": \"convention\", \"amount_bucket\": \"<10\", \"n_donations\": 945, \"total_amount\": 4660, \"mean_amount\": 4.931216931216931, \"median_amount\": 5}, {\"election_type\": \"recount\", \"amount_bucket\": \"100-500\", \"n_donations\": 2232, \"total_amount\": 413753, \"mean_amount\": 185.37320788530465, \"median_amount\": 200}, {\"election_type\": \"primary\", \"amount_bucket\": \"<10\", \"n_donations\": 2423728, \"total_amount\": 10080721, \"mean_amount\": 4.159179990493983, \"median_amount\": 5}, {\"election_type\": \"primary\", \"amount_bucket\": \"10-50\", \"n_donations\": 8115403, \"total_amount\": 187666251, \"mean_amount\": 23.12469892129818, \"median_amount\": 25}, {\"election_type\": \"primary\", \"amount_bucket\": \"50-100\", \"n_donations\": 2663933, \"total_amount\": 155426540, \"mean_amount\": 58.34476317535013, \"median_amount\": 50}, {\"election_type\": \"primary\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 684755, \"total_amount\": 1231394874, \"mean_amount\": 1798.2999379340056, \"median_amount\": 1004}, {\"election_type\": \"special\", \"amount_bucket\": \"100-500\", \"n_donations\": 34497, \"total_amount\": 5943498, \"mean_amount\": 172.29028611183583, \"median_amount\": 118}, {\"election_type\": \"convention\", \"amount_bucket\": \"100-500\", \"n_donations\": 6350, \"total_amount\": 1097843, \"mean_amount\": 172.88866141732282, \"median_amount\": 138}, {\"election_type\": \"runoff\", \"amount_bucket\": \"500-1000\", \"n_donations\": 4117, \"total_amount\": 2110393, \"mean_amount\": 512.6045664318679, \"median_amount\": 500}, {\"election_type\": \"recount\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 589, \"total_amount\": 1113150, \"mean_amount\": 1889.8981324278438, \"median_amount\": 1965}, {\"election_type\": \"recount\", \"amount_bucket\": \"10-50\", \"n_donations\": 883, \"total_amount\": 20860, \"mean_amount\": 23.62400906002265, \"median_amount\": 25}, {\"election_type\": \"recount\", \"amount_bucket\": \"50-100\", \"n_donations\": 712, \"total_amount\": 38450, \"mean_amount\": 54.002808988764045, \"median_amount\": 50}, {\"election_type\": \"recount\", \"amount_bucket\": \"<10\", \"n_donations\": 110, \"total_amount\": 569, \"mean_amount\": 5.172727272727273, \"median_amount\": 5}, {\"election_type\": \"other\", \"amount_bucket\": \"5000+\", \"n_donations\": 48, \"total_amount\": 1901300, \"mean_amount\": 39610.416666666664, \"median_amount\": 16950}], \"data-c12eefc8ce67300e6225801f6dacde98\": [{\"election_type\": \"primary\", \"amount_bucket\": \"500-1000\", \"n_donations\": 634677, \"total_amount\": 334630687, \"mean_amount\": 527.2456493618014, \"median_amount\": 500}, {\"election_type\": \"general\", \"amount_bucket\": \"5000+\", \"n_donations\": 3125, \"total_amount\": 44496373, \"mean_amount\": 14238.83936, \"median_amount\": 7527}, {\"election_type\": \"runoff\", \"amount_bucket\": \"10-50\", \"n_donations\": 20166, \"total_amount\": 461107, \"mean_amount\": 22.865565803828225, \"median_amount\": 25}, {\"election_type\": \"runoff\", \"amount_bucket\": \"50-100\", \"n_donations\": 11578, \"total_amount\": 585827, \"mean_amount\": 50.59828986007946, \"median_amount\": 50}, {\"election_type\": \"runoff\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 5196, \"total_amount\": 9601993, \"mean_amount\": 1847.958622016936, \"median_amount\": 1897}, {\"election_type\": \"recount\", \"amount_bucket\": \"500-1000\", \"n_donations\": 494, \"total_amount\": 250960, \"mean_amount\": 508.0161943319838, \"median_amount\": 500}, {\"election_type\": \"runoff\", \"amount_bucket\": \"<10\", \"n_donations\": 10191, \"total_amount\": 49621, \"mean_amount\": 4.869100186439015, \"median_amount\": 5}, {\"election_type\": \"special\", \"amount_bucket\": \"500-1000\", \"n_donations\": 7811, \"total_amount\": 4003293, \"mean_amount\": 512.5199078223019, \"median_amount\": 500}, {\"election_type\": \"runoff\", \"amount_bucket\": \"100-500\", \"n_donations\": 18193, \"total_amount\": 3088289, \"mean_amount\": 169.75149782883526, \"median_amount\": 101}, {\"election_type\": \"convention\", \"amount_bucket\": \"500-1000\", \"n_donations\": 1824, \"total_amount\": 945321, \"mean_amount\": 518.2680921052631, \"median_amount\": 500}, {\"election_type\": \"primary\", \"amount_bucket\": \"100-500\", \"n_donations\": 3636287, \"total_amount\": 637353634, \"mean_amount\": 175.27594329050484, \"median_amount\": 149}, {\"election_type\": \"special\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 7935, \"total_amount\": 13493154, \"mean_amount\": 1700.4604914933836, \"median_amount\": 1000}, {\"election_type\": \"special\", \"amount_bucket\": \"<10\", \"n_donations\": 25115, \"total_amount\": 122898, \"mean_amount\": 4.893410312562214, \"median_amount\": 5}, {\"election_type\": \"special\", \"amount_bucket\": \"10-50\", \"n_donations\": 51066, \"total_amount\": 1134616, \"mean_amount\": 22.21861904202405, \"median_amount\": 25}, {\"election_type\": \"special\", \"amount_bucket\": \"50-100\", \"n_donations\": 22859, \"total_amount\": 1177660, \"mean_amount\": 51.518439126820944, \"median_amount\": 50}, {\"election_type\": \"convention\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 2822, \"total_amount\": 4977314, \"mean_amount\": 1763.7540751240256, \"median_amount\": 1441}, {\"election_type\": null, \"amount_bucket\": \"5000+\", \"n_donations\": 48, \"total_amount\": 1622455, \"mean_amount\": 33801.145833333336, \"median_amount\": 21731}, {\"election_type\": \"recount\", \"amount_bucket\": \"100-500\", \"n_donations\": 2232, \"total_amount\": 413753, \"mean_amount\": 185.37320788530465, \"median_amount\": 200}, {\"election_type\": \"convention\", \"amount_bucket\": \"10-50\", \"n_donations\": 6848, \"total_amount\": 141604, \"mean_amount\": 20.678154205607477, \"median_amount\": 25}, {\"election_type\": \"convention\", \"amount_bucket\": \"50-100\", \"n_donations\": 2966, \"total_amount\": 153281, \"mean_amount\": 51.67936614969656, \"median_amount\": 50}, {\"election_type\": \"convention\", \"amount_bucket\": \"<10\", \"n_donations\": 945, \"total_amount\": 4660, \"mean_amount\": 4.931216931216931, \"median_amount\": 5}, {\"election_type\": \"general\", \"amount_bucket\": \"<10\", \"n_donations\": 115873, \"total_amount\": 536742, \"mean_amount\": 4.632157620843509, \"median_amount\": 5}, {\"election_type\": \"general\", \"amount_bucket\": \"10-50\", \"n_donations\": 660787, \"total_amount\": 14411588, \"mean_amount\": 21.809732939661345, \"median_amount\": 25}, {\"election_type\": \"general\", \"amount_bucket\": \"50-100\", \"n_donations\": 304363, \"total_amount\": 16184312, \"mean_amount\": 53.174374020495264, \"median_amount\": 50}, {\"election_type\": \"general\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 246101, \"total_amount\": 460025242, \"mean_amount\": 1869.2538510611496, \"median_amount\": 1970}, {\"election_type\": \"runoff\", \"amount_bucket\": \"5000+\", \"n_donations\": 37, \"total_amount\": 211400, \"mean_amount\": 5713.513513513513, \"median_amount\": 5400}, {\"election_type\": \"other\", \"amount_bucket\": \"500-1000\", \"n_donations\": 119, \"total_amount\": 62535, \"mean_amount\": 525.5042016806723, \"median_amount\": 500}, {\"election_type\": \"general\", \"amount_bucket\": \"100-500\", \"n_donations\": 700821, \"total_amount\": 123174568, \"mean_amount\": 175.75753009684357, \"median_amount\": 151}, {\"election_type\": null, \"amount_bucket\": \"500-1000\", \"n_donations\": 89, \"total_amount\": 48290, \"mean_amount\": 542.5842696629213, \"median_amount\": 500}, {\"election_type\": \"special\", \"amount_bucket\": \"5000+\", \"n_donations\": 129, \"total_amount\": 788712, \"mean_amount\": 6114.046511627907, \"median_amount\": 5400}, {\"election_type\": \"other\", \"amount_bucket\": \"100-500\", \"n_donations\": 630, \"total_amount\": 117988, \"mean_amount\": 187.2825396825397, \"median_amount\": 192}, {\"election_type\": null, \"amount_bucket\": \"1000-5000\", \"n_donations\": 116, \"total_amount\": 228657, \"mean_amount\": 1971.1810344827586, \"median_amount\": 1300}, {\"election_type\": \"convention\", \"amount_bucket\": \"5000+\", \"n_donations\": 219, \"total_amount\": 1590300, \"mean_amount\": 7261.643835616438, \"median_amount\": 8100}, {\"election_type\": null, \"amount_bucket\": \"10-50\", \"n_donations\": 151, \"total_amount\": 3167, \"mean_amount\": 20.973509933774835, \"median_amount\": 25}, {\"election_type\": null, \"amount_bucket\": \"<10\", \"n_donations\": 24, \"total_amount\": 108, \"mean_amount\": 4.5, \"median_amount\": 5}, {\"election_type\": null, \"amount_bucket\": \"50-100\", \"n_donations\": 36, \"total_amount\": 1880, \"mean_amount\": 52.22222222222222, \"median_amount\": 50}, {\"election_type\": \"primary\", \"amount_bucket\": \"10-50\", \"n_donations\": 8115403, \"total_amount\": 187666251, \"mean_amount\": 23.12469892129818, \"median_amount\": 25}, {\"election_type\": \"primary\", \"amount_bucket\": \"50-100\", \"n_donations\": 2663933, \"total_amount\": 155426540, \"mean_amount\": 58.34476317535013, \"median_amount\": 50}, {\"election_type\": \"primary\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 684755, \"total_amount\": 1231394874, \"mean_amount\": 1798.2999379340056, \"median_amount\": 1002}, {\"election_type\": \"primary\", \"amount_bucket\": \"<10\", \"n_donations\": 2423728, \"total_amount\": 10080721, \"mean_amount\": 4.159179990493983, \"median_amount\": 5}, {\"election_type\": \"special\", \"amount_bucket\": \"100-500\", \"n_donations\": 34497, \"total_amount\": 5943498, \"mean_amount\": 172.29028611183583, \"median_amount\": 119}, {\"election_type\": \"runoff\", \"amount_bucket\": \"500-1000\", \"n_donations\": 4117, \"total_amount\": 2110393, \"mean_amount\": 512.6045664318679, \"median_amount\": 500}, {\"election_type\": \"convention\", \"amount_bucket\": \"100-500\", \"n_donations\": 6350, \"total_amount\": 1097843, \"mean_amount\": 172.88866141732282, \"median_amount\": 137}, {\"election_type\": \"recount\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 589, \"total_amount\": 1113150, \"mean_amount\": 1889.8981324278438, \"median_amount\": 1965}, {\"election_type\": \"recount\", \"amount_bucket\": \"10-50\", \"n_donations\": 883, \"total_amount\": 20860, \"mean_amount\": 23.62400906002265, \"median_amount\": 25}, {\"election_type\": \"recount\", \"amount_bucket\": \"50-100\", \"n_donations\": 712, \"total_amount\": 38450, \"mean_amount\": 54.002808988764045, \"median_amount\": 50}, {\"election_type\": \"other\", \"amount_bucket\": \"5000+\", \"n_donations\": 48, \"total_amount\": 1901300, \"mean_amount\": 39610.416666666664, \"median_amount\": 16950}, {\"election_type\": \"recount\", \"amount_bucket\": \"<10\", \"n_donations\": 110, \"total_amount\": 569, \"mean_amount\": 5.172727272727273, \"median_amount\": 5}, {\"election_type\": \"general\", \"amount_bucket\": \"500-1000\", \"n_donations\": 174182, \"total_amount\": 91015697, \"mean_amount\": 522.5321617618354, \"median_amount\": 500}, {\"election_type\": \"primary\", \"amount_bucket\": \"5000+\", \"n_donations\": 44085, \"total_amount\": 1558371116, \"mean_amount\": 35349.237064761255, \"median_amount\": 10000}, {\"election_type\": \"recount\", \"amount_bucket\": \"5000+\", \"n_donations\": 26, \"total_amount\": 1888024, \"mean_amount\": 72616.30769230769, \"median_amount\": 101450}, {\"election_type\": \"other\", \"amount_bucket\": \"10-50\", \"n_donations\": 2644, \"total_amount\": 64297, \"mean_amount\": 24.318078668683814, \"median_amount\": 23}, {\"election_type\": \"other\", \"amount_bucket\": \"<10\", \"n_donations\": 10993, \"total_amount\": 25816, \"mean_amount\": 2.3484035295187846, \"median_amount\": 1}, {\"election_type\": \"other\", \"amount_bucket\": \"50-100\", \"n_donations\": 451, \"total_amount\": 27149, \"mean_amount\": 60.19733924611973, \"median_amount\": 50}, {\"election_type\": null, \"amount_bucket\": \"100-500\", \"n_donations\": 195, \"total_amount\": 46746, \"mean_amount\": 239.72307692307692, \"median_amount\": 250}, {\"election_type\": \"other\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 235, \"total_amount\": 548212, \"mean_amount\": 2332.817021276596, \"median_amount\": 2633}]}}, {\"mode\": \"vega-lite\"});\n</script>\n```\n:::\n:::\n\n\n### By election stage\n\nLet's look at how donations break down by election stage. Do people donate\ndifferently for primary elections vs general elections?\n\nLet's ignore everything but primary and general elections, since they are the\nmost common, and arguably the most important.\n\n::: {#8a758b63 .cell execution_count=25}\n``` {.python .cell-code}\ngb2 = by_type_and_bucket[_.election_type.isin((\"primary\", \"general\"))]\nn_donations_per_election_type = _.n_donations.sum().over(group_by=\"election_type\")\nfrac = _.n_donations / n_donations_per_election_type\ngb2 = gb2.mutate(frac_n_donations_per_election_type=frac)\ngb2\n```\n\n::: {.cell-output .cell-output-display execution_count=38}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> election_type </span>┃<span style=\"font-weight: bold\"> amount_bucket </span>┃<span style=\"font-weight: bold\"> n_donations </span>┃<span style=\"font-weight: bold\"> total_amount </span>┃<span style=\"font-weight: bold\"> mean_amount  </span>┃<span style=\"font-weight: bold\"> median_amount </span>┃<span style=\"font-weight: bold\"> frac_n_donations_per_election_type </span>┃\n┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>      │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>                            │\n├───────────────┼───────────────┼─────────────┼──────────────┼──────────────┼───────────────┼────────────────────────────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">&lt;10          </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115873</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">536742</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.632158</span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.052544</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">50-100       </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">304363</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">16184312</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">53.174374</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.138017</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">1000-5000    </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">246101</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">460025242</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1869.253851</span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1961</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.111598</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">10-50        </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">660787</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14411588</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21.809733</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.299642</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">700821</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">123174568</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">175.757530</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">150</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.317796</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">174182</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">91015697</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">522.532162</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.078985</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">general      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">5000+        </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3125</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">44496373</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14238.839360</span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7601</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.001417</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">5000+        </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">44085</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1558371116</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35349.237065</span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10000</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.002422</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">100-500      </span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3636287</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">637353634</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">175.275943</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">150</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.199765</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">primary      </span> │ <span style=\"color: #008000; text-decoration-color: #008000\">500-1000     </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">634677</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">334630687</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">527.245649</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> │                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.034867</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>             │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└───────────────┴───────────────┴─────────────┴──────────────┴──────────────┴───────────────┴────────────────────────────────────┘\n</pre>\n```\n:::\n:::\n\n\nIt looks like primary elections get a larger proportion of small donations.\n\n::: {#30710ce2 .cell execution_count=26}\n``` {.python .cell-code}\nalt.Chart(gb2.execute()).mark_bar().encode(\n    x=\"election_type:O\",\n    y=\"frac_n_donations_per_election_type:Q\",\n    color=bucket_col,\n)\n```\n\n::: {.cell-output .cell-output-display execution_count=39}\n```{=html}\n\n<style>\n  #altair-viz-c7d7cb33ea8c45b6bbd9679963751c34.vega-embed {\n    width: 100%;\n    display: flex;\n  }\n\n  #altair-viz-c7d7cb33ea8c45b6bbd9679963751c34.vega-embed details,\n  #altair-viz-c7d7cb33ea8c45b6bbd9679963751c34.vega-embed details summary {\n    position: relative;\n  }\n</style>\n<div id=\"altair-viz-c7d7cb33ea8c45b6bbd9679963751c34\"></div>\n<script type=\"text/javascript\">\n  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n  (function(spec, embedOpt){\n    let outputDiv = document.currentScript.previousElementSibling;\n    if (outputDiv.id !== \"altair-viz-c7d7cb33ea8c45b6bbd9679963751c34\") {\n      outputDiv = document.getElementById(\"altair-viz-c7d7cb33ea8c45b6bbd9679963751c34\");\n    }\n    const paths = {\n      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.16.3?noext\",\n      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n    };\n\n    function maybeLoadScript(lib, version) {\n      var key = `${lib.replace(\"-\", \"\")}_version`;\n      return (VEGA_DEBUG[key] == version) ?\n        Promise.resolve(paths[lib]) :\n        new Promise(function(resolve, reject) {\n          var s = document.createElement('script');\n          document.getElementsByTagName(\"head\")[0].appendChild(s);\n          s.async = true;\n          s.onload = () => {\n            VEGA_DEBUG[key] = version;\n            return resolve(paths[lib]);\n          };\n          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n          s.src = paths[lib];\n        });\n    }\n\n    function showError(err) {\n      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n      throw err;\n    }\n\n    function displayChart(vegaEmbed) {\n      vegaEmbed(outputDiv, spec, embedOpt)\n        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n    }\n\n    if(typeof define === \"function\" && define.amd) {\n      requirejs.config({paths});\n      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n    } else {\n      maybeLoadScript(\"vega\", \"5\")\n        .then(() => maybeLoadScript(\"vega-lite\", \"5.16.3\"))\n        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n        .catch(showError)\n        .then(() => displayChart(vegaEmbed));\n    }\n  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-ebbd3804e28f31fe1e149c3016fe9de2\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"color\": {\"field\": \"amount_bucket\", \"sort\": [\"<10\", \"10-50\", \"50-100\", \"100-500\", \"500-1000\", \"1000-5000\", \"5000+\"], \"type\": \"nominal\"}, \"x\": {\"field\": \"election_type\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"frac_n_donations_per_election_type\", \"type\": \"quantitative\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.16.3.json\", \"datasets\": {\"data-ebbd3804e28f31fe1e149c3016fe9de2\": [{\"election_type\": \"general\", \"amount_bucket\": \"500-1000\", \"n_donations\": 174182, \"total_amount\": 91015697, \"mean_amount\": 522.5321617618354, \"median_amount\": 500, \"frac_n_donations_per_election_type\": 0.0789850774423966}, {\"election_type\": \"general\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 246101, \"total_amount\": 460025242, \"mean_amount\": 1869.2538510611496, \"median_amount\": 1962, \"frac_n_donations_per_election_type\": 0.1115976768187944}, {\"election_type\": \"general\", \"amount_bucket\": \"50-100\", \"n_donations\": 304363, \"total_amount\": 16184312, \"mean_amount\": 53.174374020495264, \"median_amount\": 50, \"frac_n_donations_per_election_type\": 0.13801733316645898}, {\"election_type\": \"general\", \"amount_bucket\": \"<10\", \"n_donations\": 115873, \"total_amount\": 536742, \"mean_amount\": 4.632157620843509, \"median_amount\": 5, \"frac_n_donations_per_election_type\": 0.052544108337731925}, {\"election_type\": \"general\", \"amount_bucket\": \"10-50\", \"n_donations\": 660787, \"total_amount\": 14411588, \"mean_amount\": 21.809732939661345, \"median_amount\": 25, \"frac_n_donations_per_election_type\": 0.2996423991453131}, {\"election_type\": \"general\", \"amount_bucket\": \"5000+\", \"n_donations\": 3125, \"total_amount\": 44496373, \"mean_amount\": 14238.83936, \"median_amount\": 7534, \"frac_n_donations_per_election_type\": 0.0014170716090496688}, {\"election_type\": \"general\", \"amount_bucket\": \"100-500\", \"n_donations\": 700821, \"total_amount\": 123174568, \"mean_amount\": 175.75753009684357, \"median_amount\": 150, \"frac_n_donations_per_election_type\": 0.3177963334802553}, {\"election_type\": \"primary\", \"amount_bucket\": \"1000-5000\", \"n_donations\": 684755, \"total_amount\": 1231394874, \"mean_amount\": 1798.2999379340056, \"median_amount\": 1005, \"frac_n_donations_per_election_type\": 0.037617973167744775}, {\"election_type\": \"primary\", \"amount_bucket\": \"50-100\", \"n_donations\": 2663933, \"total_amount\": 155426540, \"mean_amount\": 58.34476317535013, \"median_amount\": 50, \"frac_n_donations_per_election_type\": 0.14634688335925966}, {\"election_type\": \"primary\", \"amount_bucket\": \"<10\", \"n_donations\": 2423728, \"total_amount\": 10080721, \"mean_amount\": 4.159179990493983, \"median_amount\": 5, \"frac_n_donations_per_election_type\": 0.13315088589336582}, {\"election_type\": \"primary\", \"amount_bucket\": \"10-50\", \"n_donations\": 8115403, \"total_amount\": 187666251, \"mean_amount\": 23.12469892129818, \"median_amount\": 25, \"frac_n_donations_per_election_type\": 0.44583100860809405}, {\"election_type\": \"primary\", \"amount_bucket\": \"5000+\", \"n_donations\": 44085, \"total_amount\": 1558371116, \"mean_amount\": 35349.237064761255, \"median_amount\": 10000, \"frac_n_donations_per_election_type\": 0.0024218711029492714}, {\"election_type\": \"primary\", \"amount_bucket\": \"500-1000\", \"n_donations\": 634677, \"total_amount\": 334630687, \"mean_amount\": 527.2456493618014, \"median_amount\": 500, \"frac_n_donations_per_election_type\": 0.03486686823197312}, {\"election_type\": \"primary\", \"amount_bucket\": \"100-500\", \"n_donations\": 3636287, \"total_amount\": 637353634, \"mean_amount\": 175.27594329050484, \"median_amount\": 150, \"frac_n_donations_per_election_type\": 0.1997645096366133}]}}, {\"mode\": \"vega-lite\"});\n</script>\n```\n:::\n:::\n\n\n### By recipient\n\nLet's look at the top players. Who gets the most donations?\n\nFar and away it is ActBlue, which acts as a conduit for donations to Democratic\ninterests.\n\nBeto O'Rourke is the top individual politician, hats off to him!\n\n::: {#97c0a2c8 .cell execution_count=27}\n``` {.python .cell-code}\nby_recip = summary_by(featured, \"CMTE_NM\")\nby_recip\n```\n\n::: {.cell-output .cell-output-display execution_count=40}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> CMTE_NM                                                          </span>┃<span style=\"font-weight: bold\"> n_donations </span>┃<span style=\"font-weight: bold\"> total_amount </span>┃<span style=\"font-weight: bold\"> mean_amount </span>┃<span style=\"font-weight: bold\"> median_amount </span>┃\n┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>                                                           │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>         │\n├──────────────────────────────────────────────────────────────────┼─────────────┼──────────────┼─────────────┼───────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">INDIANA DENTAL PAC                                              </span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">62236</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">560.684685</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">410</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">BEAM SUNTORY INC POLITICAL ACTION COMMITTEE                     </span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">407</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">64806</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">159.228501</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">65</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">AMEDISYS, INC. POLITICAL ACTION COMMITTEE                       </span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">132</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25000</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">189.393939</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">75</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">PIEDMONT TRIAD ANESTHESIA P A FEDERAL PAC                       </span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">132</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">90375</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">684.659091</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">600</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">AHOLD DELHAIZE USA, INC POLITICAL ACTION COMMITTEE              </span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">369</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">48062</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">130.249322</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">DIMITRI FOR CONGRESS                                            </span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">87</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">34719</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">399.068966</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">250</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">RELX INC. POLITICAL ACTION COMMITTEE                            </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5491</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">306908</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">55.892916</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">34</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">MAKING INVESTMENTS MAJORITY INSURED PAC                         </span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">30600</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2185.714286</span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1000</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">AMERICAN ACADEMY OF OTOLARYNGOLOGY-HEAD AND NECK SURGERY ENT PAC</span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">765</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">285756</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">373.537255</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">365</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">MIMI WALTERS VICTORY FUND                                       </span> │         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">840</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2514824</span> │ <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2993.838095</span> │          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2506</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>                                                                │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└──────────────────────────────────────────────────────────────────┴─────────────┴──────────────┴─────────────┴───────────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#56418e6e .cell execution_count=28}\n``` {.python .cell-code}\ntop_recip = by_recip.order_by(ibis.desc(\"n_donations\")).head(10)\nalt.Chart(top_recip.execute()).mark_bar().encode(\n    x=alt.X(\"CMTE_NM:O\", sort=\"-y\"),\n    y=\"n_donations:Q\",\n)\n```\n\n::: {.cell-output .cell-output-display execution_count=41}\n```{=html}\n\n<style>\n  #altair-viz-22b463e147124dfdb214c1ccc86159eb.vega-embed {\n    width: 100%;\n    display: flex;\n  }\n\n  #altair-viz-22b463e147124dfdb214c1ccc86159eb.vega-embed details,\n  #altair-viz-22b463e147124dfdb214c1ccc86159eb.vega-embed details summary {\n    position: relative;\n  }\n</style>\n<div id=\"altair-viz-22b463e147124dfdb214c1ccc86159eb\"></div>\n<script type=\"text/javascript\">\n  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n  (function(spec, embedOpt){\n    let outputDiv = document.currentScript.previousElementSibling;\n    if (outputDiv.id !== \"altair-viz-22b463e147124dfdb214c1ccc86159eb\") {\n      outputDiv = document.getElementById(\"altair-viz-22b463e147124dfdb214c1ccc86159eb\");\n    }\n    const paths = {\n      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.16.3?noext\",\n      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n    };\n\n    function maybeLoadScript(lib, version) {\n      var key = `${lib.replace(\"-\", \"\")}_version`;\n      return (VEGA_DEBUG[key] == version) ?\n        Promise.resolve(paths[lib]) :\n        new Promise(function(resolve, reject) {\n          var s = document.createElement('script');\n          document.getElementsByTagName(\"head\")[0].appendChild(s);\n          s.async = true;\n          s.onload = () => {\n            VEGA_DEBUG[key] = version;\n            return resolve(paths[lib]);\n          };\n          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n          s.src = paths[lib];\n        });\n    }\n\n    function showError(err) {\n      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n      throw err;\n    }\n\n    function displayChart(vegaEmbed) {\n      vegaEmbed(outputDiv, spec, embedOpt)\n        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n    }\n\n    if(typeof define === \"function\" && define.amd) {\n      requirejs.config({paths});\n      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n    } else {\n      maybeLoadScript(\"vega\", \"5\")\n        .then(() => maybeLoadScript(\"vega-lite\", \"5.16.3\"))\n        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n        .catch(showError)\n        .then(() => displayChart(vegaEmbed));\n    }\n  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-ce10b3f5b7c7e35451245a008d469163\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"CMTE_NM\", \"sort\": \"-y\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"n_donations\", \"type\": \"quantitative\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.16.3.json\", \"datasets\": {\"data-ce10b3f5b7c7e35451245a008d469163\": [{\"CMTE_NM\": \"ACTBLUE\", \"n_donations\": 5820888, \"total_amount\": 693057213, \"mean_amount\": 119.06382892094814, \"median_amount\": 25}, {\"CMTE_NM\": \"DCCC\", \"n_donations\": 1315476, \"total_amount\": 124802082, \"mean_amount\": 94.87218466927561, \"median_amount\": 25}, {\"CMTE_NM\": \"REPUBLICAN NATIONAL COMMITTEE\", \"n_donations\": 570561, \"total_amount\": 131525422, \"mean_amount\": 230.5194746924518, \"median_amount\": 50}, {\"CMTE_NM\": \"END CITIZENS UNITED\", \"n_donations\": 489710, \"total_amount\": 13654987, \"mean_amount\": 27.8838230789651, \"median_amount\": 15}, {\"CMTE_NM\": \"DSCC\", \"n_donations\": 347493, \"total_amount\": 67844824, \"mean_amount\": 195.2408365060591, \"median_amount\": 35}, {\"CMTE_NM\": \"PROGRESSIVE TURNOUT PROJECT\", \"n_donations\": 313433, \"total_amount\": 9251647, \"mean_amount\": 29.517144014829327, \"median_amount\": 15}, {\"CMTE_NM\": \"DNC SERVICES CORP./DEM. NAT'L COMMITTEE\", \"n_donations\": 280264, \"total_amount\": 70156788, \"mean_amount\": 250.32393743042275, \"median_amount\": 50}, {\"CMTE_NM\": \"BETO FOR TEXAS\", \"n_donations\": 280027, \"total_amount\": 44914966, \"mean_amount\": 160.39512618426082, \"median_amount\": 50}, {\"CMTE_NM\": \"NRSC\", \"n_donations\": 203124, \"total_amount\": 55384644, \"mean_amount\": 272.66420511608675, \"median_amount\": 50}, {\"CMTE_NM\": \"NRCC\", \"n_donations\": 178176, \"total_amount\": 38646560, \"mean_amount\": 216.90104166666666, \"median_amount\": 50}]}}, {\"mode\": \"vega-lite\"});\n</script>\n```\n:::\n:::\n\n\n### By Location\n\nWhere are the largest donations coming from?\n\n::: {#55b19fc3 .cell execution_count=29}\n``` {.python .cell-code}\nf2 = featured.mutate(loc=_.CITY + \", \" + _.STATE).drop(\"CITY\", \"STATE\")\nby_loc = summary_by(f2, \"loc\")\n# Drop the places with a small number of donations so we're\n# resistant to outliers for the mean\nby_loc = by_loc[_.n_donations > 1000]\nby_loc\n```\n\n::: {.cell-output .cell-output-display execution_count=42}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> loc             </span>┃<span style=\"font-weight: bold\"> n_donations </span>┃<span style=\"font-weight: bold\"> total_amount </span>┃<span style=\"font-weight: bold\"> mean_amount </span>┃<span style=\"font-weight: bold\"> median_amount </span>┃\n┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>          │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>         │\n├─────────────────┼─────────────┼──────────────┼─────────────┼───────────────┤\n│ <span style=\"color: #008000; text-decoration-color: #008000\">NAZARETH, PA   </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1460</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">138710</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">95.006849</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">38</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">FULSHEAR, TX   </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1504</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">346778</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">230.570479</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">GLOUCESTER, MA </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4956</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">563331</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">113.666465</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">NORMAN, OK     </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6195</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">945333</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">152.596126</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">OAK PARK, IL   </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">12017</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3413138</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">284.025797</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">39</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">AUSTIN, TX     </span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">189865</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">33315922</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">175.471635</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">38</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">MIAMI BEACH, FL</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">12825</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10598453</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">826.390097</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">SAN ANTONIO, TX</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">140529</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18925978</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">134.676672</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">HAMBURG, NY    </span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2322</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">170254</span> │   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">73.322136</span> │             <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span> │\n│ <span style=\"color: #008000; text-decoration-color: #008000\">PITTSBURGH, PA </span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">74208</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14358578</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">193.490971</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">42</span> │\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>               │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │\n└─────────────────┴─────────────┴──────────────┴─────────────┴───────────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#cc1697c5 .cell execution_count=30}\n``` {.python .cell-code}\ndef top_by(col):\n    top = by_loc.order_by(ibis.desc(col)).head(10)\n    return (\n        alt.Chart(top.execute())\n        .mark_bar()\n        .encode(\n            x=alt.X('loc:O', sort=\"-y\"),\n            y=col,\n        )\n    )\n\n\ntop_by(\"n_donations\") | top_by(\"total_amount\") | top_by(\"mean_amount\") | top_by(\n    \"median_amount\"\n)\n```\n\n::: {.cell-output .cell-output-display execution_count=43}\n```{=html}\n\n<style>\n  #altair-viz-8a6f1f32899f46cc9b2d62e6535d25ef.vega-embed {\n    width: 100%;\n    display: flex;\n  }\n\n  #altair-viz-8a6f1f32899f46cc9b2d62e6535d25ef.vega-embed details,\n  #altair-viz-8a6f1f32899f46cc9b2d62e6535d25ef.vega-embed details summary {\n    position: relative;\n  }\n</style>\n<div id=\"altair-viz-8a6f1f32899f46cc9b2d62e6535d25ef\"></div>\n<script type=\"text/javascript\">\n  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n  (function(spec, embedOpt){\n    let outputDiv = document.currentScript.previousElementSibling;\n    if (outputDiv.id !== \"altair-viz-8a6f1f32899f46cc9b2d62e6535d25ef\") {\n      outputDiv = document.getElementById(\"altair-viz-8a6f1f32899f46cc9b2d62e6535d25ef\");\n    }\n    const paths = {\n      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.16.3?noext\",\n      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n    };\n\n    function maybeLoadScript(lib, version) {\n      var key = `${lib.replace(\"-\", \"\")}_version`;\n      return (VEGA_DEBUG[key] == version) ?\n        Promise.resolve(paths[lib]) :\n        new Promise(function(resolve, reject) {\n          var s = document.createElement('script');\n          document.getElementsByTagName(\"head\")[0].appendChild(s);\n          s.async = true;\n          s.onload = () => {\n            VEGA_DEBUG[key] = version;\n            return resolve(paths[lib]);\n          };\n          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n          s.src = paths[lib];\n        });\n    }\n\n    function showError(err) {\n      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n      throw err;\n    }\n\n    function displayChart(vegaEmbed) {\n      vegaEmbed(outputDiv, spec, embedOpt)\n        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n    }\n\n    if(typeof define === \"function\" && define.amd) {\n      requirejs.config({paths});\n      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n    } else {\n      maybeLoadScript(\"vega\", \"5\")\n        .then(() => maybeLoadScript(\"vega-lite\", \"5.16.3\"))\n        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n        .catch(showError)\n        .then(() => displayChart(vegaEmbed));\n    }\n  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"hconcat\": [{\"data\": {\"name\": \"data-88a0fd8958c48a49df7689732aa79f72\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"loc\", \"sort\": \"-y\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"n_donations\", \"type\": \"quantitative\"}}}, {\"data\": {\"name\": \"data-63253180effd0e4c1cba64f44dc0588d\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"loc\", \"sort\": \"-y\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"total_amount\", \"type\": \"quantitative\"}}}, {\"data\": {\"name\": \"data-0053131ee6866911d91bd779adba39b7\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"loc\", \"sort\": \"-y\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"mean_amount\", \"type\": \"quantitative\"}}}, {\"data\": {\"name\": \"data-c6410b8d1e323aef90b691e51bbd6e4d\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"loc\", \"sort\": \"-y\", \"type\": \"ordinal\"}, \"y\": {\"field\": \"median_amount\", \"type\": \"quantitative\"}}}], \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.16.3.json\", \"datasets\": {\"data-88a0fd8958c48a49df7689732aa79f72\": [{\"loc\": \"NEW YORK, NY\", \"n_donations\": 695091, \"total_amount\": 444600108, \"mean_amount\": 639.6286356750411, \"median_amount\": 50}, {\"loc\": \"WASHINGTON, DC\", \"n_donations\": 401498, \"total_amount\": 124456508, \"mean_amount\": 309.9803934266173, \"median_amount\": 57}, {\"loc\": \"HOUSTON, TX\", \"n_donations\": 251960, \"total_amount\": 83026989, \"mean_amount\": 329.5244840450865, \"median_amount\": 50}, {\"loc\": \"LOS ANGELES, CA\", \"n_donations\": 245263, \"total_amount\": 89881980, \"mean_amount\": 366.4718282007478, \"median_amount\": 50}, {\"loc\": \"SAN FRANCISCO, CA\", \"n_donations\": 238117, \"total_amount\": 189799961, \"mean_amount\": 797.086982449804, \"median_amount\": 50}, {\"loc\": \"PHILADELPHIA, PA\", \"n_donations\": 222938, \"total_amount\": 36054977, \"mean_amount\": 161.72647552234253, \"median_amount\": 62}, {\"loc\": \"CHICAGO, IL\", \"n_donations\": 212527, \"total_amount\": 108119674, \"mean_amount\": 508.7338267608351, \"median_amount\": 40}, {\"loc\": \"SEATTLE, WA\", \"n_donations\": 197671, \"total_amount\": 52867387, \"mean_amount\": 267.4514066302088, \"median_amount\": 36}, {\"loc\": \"AUSTIN, TX\", \"n_donations\": 189865, \"total_amount\": 33315922, \"mean_amount\": 175.4716351091565, \"median_amount\": 38}, {\"loc\": \"ARLINGTON, VA\", \"n_donations\": 163168, \"total_amount\": 23382868, \"mean_amount\": 143.30547656403218, \"median_amount\": 50}], \"data-63253180effd0e4c1cba64f44dc0588d\": [{\"loc\": \"NEW YORK, NY\", \"n_donations\": 695091, \"total_amount\": 444600108, \"mean_amount\": 639.6286356750411, \"median_amount\": 50}, {\"loc\": \"SAN FRANCISCO, CA\", \"n_donations\": 238117, \"total_amount\": 189799961, \"mean_amount\": 797.086982449804, \"median_amount\": 50}, {\"loc\": \"LAS VEGAS, NV\", \"n_donations\": 65940, \"total_amount\": 153467387, \"mean_amount\": 2327.37923870185, \"median_amount\": 46}, {\"loc\": \"WASHINGTON, DC\", \"n_donations\": 401498, \"total_amount\": 124456508, \"mean_amount\": 309.9803934266173, \"median_amount\": 57}, {\"loc\": \"CHICAGO, IL\", \"n_donations\": 212527, \"total_amount\": 108119674, \"mean_amount\": 508.7338267608351, \"median_amount\": 40}, {\"loc\": \"LOS ANGELES, CA\", \"n_donations\": 245263, \"total_amount\": 89881980, \"mean_amount\": 366.4718282007478, \"median_amount\": 50}, {\"loc\": \"HOUSTON, TX\", \"n_donations\": 251960, \"total_amount\": 83026989, \"mean_amount\": 329.5244840450865, \"median_amount\": 50}, {\"loc\": \"DALLAS, TX\", \"n_donations\": 154038, \"total_amount\": 66558403, \"mean_amount\": 432.09080226956985, \"median_amount\": 57}, {\"loc\": \"SEATTLE, WA\", \"n_donations\": 197671, \"total_amount\": 52867387, \"mean_amount\": 267.4514066302088, \"median_amount\": 36}, {\"loc\": \"BOSTON, MA\", \"n_donations\": 82925, \"total_amount\": 47592049, \"mean_amount\": 573.9167802230932, \"median_amount\": 58}], \"data-0053131ee6866911d91bd779adba39b7\": [{\"loc\": \"LAKE FOREST, IL\", \"n_donations\": 5636, \"total_amount\": 37486362, \"mean_amount\": 6651.235273243435, \"median_amount\": 100}, {\"loc\": \"MOUNT VERNON, OH\", \"n_donations\": 1431, \"total_amount\": 5605857, \"mean_amount\": 3917.4402515723273, \"median_amount\": 46}, {\"loc\": \"LOS ALTOS HILLS, CA\", \"n_donations\": 4098, \"total_amount\": 10367629, \"mean_amount\": 2529.92410932162, \"median_amount\": 326}, {\"loc\": \"PALM BEACH, FL\", \"n_donations\": 7140, \"total_amount\": 17212425, \"mean_amount\": 2410.703781512605, \"median_amount\": 255}, {\"loc\": \"LAS VEGAS, NV\", \"n_donations\": 65940, \"total_amount\": 153467387, \"mean_amount\": 2327.37923870185, \"median_amount\": 45}, {\"loc\": \"RHINEBECK, NY\", \"n_donations\": 3014, \"total_amount\": 5942571, \"mean_amount\": 1971.6559389515594, \"median_amount\": 46}, {\"loc\": \"JOPLIN, MO\", \"n_donations\": 1839, \"total_amount\": 3617186, \"mean_amount\": 1966.9309407286569, \"median_amount\": 50}, {\"loc\": \"BALA CYNWYD, PA\", \"n_donations\": 3668, \"total_amount\": 6949933, \"mean_amount\": 1894.7472737186479, \"median_amount\": 100}, {\"loc\": \"CARMEL, IN\", \"n_donations\": 10932, \"total_amount\": 20383688, \"mean_amount\": 1864.5890962312478, \"median_amount\": 53}, {\"loc\": \"WAYLAND, MA\", \"n_donations\": 5283, \"total_amount\": 9704279, \"mean_amount\": 1836.8879424569373, \"median_amount\": 50}], \"data-c6410b8d1e323aef90b691e51bbd6e4d\": [{\"loc\": \"GLADWYNE, PA\", \"n_donations\": 1727, \"total_amount\": 1333243, \"mean_amount\": 771.9994209612044, \"median_amount\": 337}, {\"loc\": \"LOS ALTOS HILLS, CA\", \"n_donations\": 4098, \"total_amount\": 10367629, \"mean_amount\": 2529.92410932162, \"median_amount\": 313}, {\"loc\": \"MC LEAN, VA\", \"n_donations\": 4692, \"total_amount\": 3656109, \"mean_amount\": 779.2218670076726, \"median_amount\": 307}, {\"loc\": \"PALM BEACH, FL\", \"n_donations\": 7140, \"total_amount\": 17212425, \"mean_amount\": 2410.703781512605, \"median_amount\": 255}, {\"loc\": \"MISSION HILLS, KS\", \"n_donations\": 2258, \"total_amount\": 1642339, \"mean_amount\": 727.3423383525244, \"median_amount\": 250}, {\"loc\": \"DOVER, MA\", \"n_donations\": 1040, \"total_amount\": 976757, \"mean_amount\": 939.189423076923, \"median_amount\": 250}, {\"loc\": \"SHORT HILLS, NJ\", \"n_donations\": 3555, \"total_amount\": 3396742, \"mean_amount\": 955.4829817158931, \"median_amount\": 250}, {\"loc\": \"PARADISE VALLEY, AZ\", \"n_donations\": 8197, \"total_amount\": 7035291, \"mean_amount\": 858.2763206050994, \"median_amount\": 250}, {\"loc\": \"ATHERTON, CA\", \"n_donations\": 8780, \"total_amount\": 11595391, \"mean_amount\": 1320.6595671981777, \"median_amount\": 250}, {\"loc\": \"KENILWORTH, IL\", \"n_donations\": 1500, \"total_amount\": 855723, \"mean_amount\": 570.482, \"median_amount\": 250}]}}, {\"mode\": \"vega-lite\"});\n</script>\n```\n:::\n:::\n\n\n### By month\n\nWhen do the donations come in?\n\n::: {#0d055d90 .cell execution_count=31}\n``` {.python .cell-code}\nby_month = summary_by(featured, _.date.month().name(\"month_int\"))\n# Sorta hacky, .substritute doesn't work to change dtypes (yet?)\n# so we cast to string and then do our mapping\nmonth_map = {\n    \"1\": \"Jan\",\n    \"2\": \"Feb\",\n    \"3\": \"Mar\",\n    \"4\": \"Apr\",\n    \"5\": \"May\",\n    \"6\": \"Jun\",\n    \"7\": \"Jul\",\n    \"8\": \"Aug\",\n    \"9\": \"Sep\",\n    \"10\": \"Oct\",\n    \"11\": \"Nov\",\n    \"12\": \"Dec\",\n}\nby_month = by_month.mutate(month_str=_.month_int.cast(str).substitute(month_map))\nby_month\n```\n\n::: {.cell-output .cell-output-display execution_count=44}\n```{=html}\n<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓\n┃<span style=\"font-weight: bold\"> month_int </span>┃<span style=\"font-weight: bold\"> n_donations </span>┃<span style=\"font-weight: bold\"> total_amount </span>┃<span style=\"font-weight: bold\"> mean_amount </span>┃<span style=\"font-weight: bold\"> median_amount </span>┃<span style=\"font-weight: bold\"> month_str </span>┃\n┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩\n│ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int32</span>     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>       │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>        │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">float64</span>     │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">int64</span>         │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">string</span>    │\n├───────────┼─────────────┼──────────────┼─────────────┼───────────────┼───────────┤\n│      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span> │        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1514</span> │       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">250297</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">165.321664</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">NULL</span>      │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">348979</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">174837854</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500.998209</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">124</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Jan      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span> │      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">581646</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">255997655</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">440.126219</span> │           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Feb      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1042577</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">430906797</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">413.309326</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">81</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Mar      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1088244</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">299252692</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">274.986760</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Apr      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1374247</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">387317192</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">281.839576</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">48</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">May      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1667285</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">465305247</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">279.079610</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">44</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Jun      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1607053</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">320528605</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">199.451172</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Jul      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2023466</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">473544182</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">234.026261</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">35</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Aug      </span> │\n│         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span> │     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2583847</span> │    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">697888624</span> │  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">270.096729</span> │            <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">38</span> │ <span style=\"color: #008000; text-decoration-color: #008000\">Sep      </span> │\n│         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span> │ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">…</span>         │\n└───────────┴─────────────┴──────────────┴─────────────┴───────────────┴───────────┘\n</pre>\n```\n:::\n:::\n\n\n::: {#7002ddb8 .cell execution_count=32}\n``` {.python .cell-code}\nmonths_in_order = list(month_map.values())\nalt.Chart(by_month.execute()).mark_bar().encode(\n    x=alt.X(\"month_str:O\", sort=months_in_order),\n    y=\"n_donations:Q\",\n)\n```\n\n::: {.cell-output .cell-output-display execution_count=45}\n```{=html}\n\n<style>\n  #altair-viz-fd0cec36d6a04996a5559215e6fce4df.vega-embed {\n    width: 100%;\n    display: flex;\n  }\n\n  #altair-viz-fd0cec36d6a04996a5559215e6fce4df.vega-embed details,\n  #altair-viz-fd0cec36d6a04996a5559215e6fce4df.vega-embed details summary {\n    position: relative;\n  }\n</style>\n<div id=\"altair-viz-fd0cec36d6a04996a5559215e6fce4df\"></div>\n<script type=\"text/javascript\">\n  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n  (function(spec, embedOpt){\n    let outputDiv = document.currentScript.previousElementSibling;\n    if (outputDiv.id !== \"altair-viz-fd0cec36d6a04996a5559215e6fce4df\") {\n      outputDiv = document.getElementById(\"altair-viz-fd0cec36d6a04996a5559215e6fce4df\");\n    }\n    const paths = {\n      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.16.3?noext\",\n      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n    };\n\n    function maybeLoadScript(lib, version) {\n      var key = `${lib.replace(\"-\", \"\")}_version`;\n      return (VEGA_DEBUG[key] == version) ?\n        Promise.resolve(paths[lib]) :\n        new Promise(function(resolve, reject) {\n          var s = document.createElement('script');\n          document.getElementsByTagName(\"head\")[0].appendChild(s);\n          s.async = true;\n          s.onload = () => {\n            VEGA_DEBUG[key] = version;\n            return resolve(paths[lib]);\n          };\n          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n          s.src = paths[lib];\n        });\n    }\n\n    function showError(err) {\n      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n      throw err;\n    }\n\n    function displayChart(vegaEmbed) {\n      vegaEmbed(outputDiv, spec, embedOpt)\n        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n    }\n\n    if(typeof define === \"function\" && define.amd) {\n      requirejs.config({paths});\n      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n    } else {\n      maybeLoadScript(\"vega\", \"5\")\n        .then(() => maybeLoadScript(\"vega-lite\", \"5.16.3\"))\n        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n        .catch(showError)\n        .then(() => displayChart(vegaEmbed));\n    }\n  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-f45f98a3a06c83bbafc4d3d94cb29c0b\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"x\": {\"field\": \"month_str\", \"sort\": [\"Jan\", \"Feb\", \"Mar\", \"Apr\", \"May\", \"Jun\", \"Jul\", \"Aug\", \"Sep\", \"Oct\", \"Nov\", \"Dec\"], \"type\": \"ordinal\"}, \"y\": {\"field\": \"n_donations\", \"type\": \"quantitative\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.16.3.json\", \"datasets\": {\"data-f45f98a3a06c83bbafc4d3d94cb29c0b\": [{\"month_int\": null, \"n_donations\": 1514, \"total_amount\": 250297, \"mean_amount\": 165.3216644649934, \"median_amount\": 100, \"month_str\": null}, {\"month_int\": 1.0, \"n_donations\": 348979, \"total_amount\": 174837854, \"mean_amount\": 500.9982090612902, \"median_amount\": 122, \"month_str\": \"Jan\"}, {\"month_int\": 2.0, \"n_donations\": 581646, \"total_amount\": 255997655, \"mean_amount\": 440.126219384299, \"median_amount\": 100, \"month_str\": \"Feb\"}, {\"month_int\": 3.0, \"n_donations\": 1042577, \"total_amount\": 430906797, \"mean_amount\": 413.3093258339672, \"median_amount\": 80, \"month_str\": \"Mar\"}, {\"month_int\": 4.0, \"n_donations\": 1088244, \"total_amount\": 299252692, \"mean_amount\": 274.98676032213365, \"median_amount\": 50, \"month_str\": \"Apr\"}, {\"month_int\": 5.0, \"n_donations\": 1374247, \"total_amount\": 387317192, \"mean_amount\": 281.83957614606396, \"median_amount\": 48, \"month_str\": \"May\"}, {\"month_int\": 6.0, \"n_donations\": 1667285, \"total_amount\": 465305247, \"mean_amount\": 279.07960966481437, \"median_amount\": 44, \"month_str\": \"Jun\"}, {\"month_int\": 7.0, \"n_donations\": 1607053, \"total_amount\": 320528605, \"mean_amount\": 199.45117242555162, \"median_amount\": 35, \"month_str\": \"Jul\"}, {\"month_int\": 8.0, \"n_donations\": 2023466, \"total_amount\": 473544182, \"mean_amount\": 234.02626088108227, \"median_amount\": 35, \"month_str\": \"Aug\"}, {\"month_int\": 9.0, \"n_donations\": 2583847, \"total_amount\": 697888624, \"mean_amount\": 270.0967294116099, \"median_amount\": 38, \"month_str\": \"Sep\"}, {\"month_int\": 10.0, \"n_donations\": 3686024, \"total_amount\": 850820707, \"mean_amount\": 230.82343115508743, \"median_amount\": 29, \"month_str\": \"Oct\"}, {\"month_int\": 11.0, \"n_donations\": 2545616, \"total_amount\": 285143995, \"mean_amount\": 112.01375030640914, \"median_amount\": 25, \"month_str\": \"Nov\"}, {\"month_int\": 12.0, \"n_donations\": 2119311, \"total_amount\": 283081648, \"mean_amount\": 133.57249030463203, \"median_amount\": 25, \"month_str\": \"Dec\"}]}}, {\"mode\": \"vega-lite\"});\n</script>\n```\n:::\n:::\n\n\n## Conclusion\n\nThanks for following along! I hope you've learned something about Ibis, and\nmaybe even about campaign finance.\n\nIbis is a great tool for exploring data. I now find myself reaching for it\nwhen in the past I would have reached for pandas.\n\nSome of the highlights for me:\n\n- Fast, lazy execution, a great display format, and good type hinting/editor support for a great REPL experience.\n- Very well thought-out API and semantics (e.g. `isinstance(val, NumericValue)`?? That's beautiful!)\n- Fast and fairly complete string support, since I work with a lot of text data.\n- Extremely responsive maintainers. Sometimes I've submitted multiple feature requests and bug reports in a single day, and a PR has been merged by the next day.\n- Escape hatch to SQL. I didn't have to use that here, but if something isn't supported, you can always fall back to SQL.\n\nCheck out [The Ibis Website](https://ibis-project.org/) for more information.\n\n",
     "supporting": [
-      "index_files/figure-html"
+      "index_files"
     ],
     "filters": [],
     "includes": {
       "include-in-header": [
-        "<script src=\"https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js\" integrity=\"sha512-c3Nl8+7g4LMSTdrm621y7kf9v3SDPnhxLNhcjFJbKECVnmZHTdo+IRO05sNLTH/D3vA6u1X32ehoLC7WFVdheg==\" crossorigin=\"anonymous\"></script>\n<script src=\"https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js\" integrity=\"sha512-bLT0Qm9VnAYZDflyKcBaQ2gg0hSYNQrJ8RilYldYQ1FxQYoCLtUjuuRuZo+fjqhx/qtq/1itJ0C2ejDxltZVFg==\" crossorigin=\"anonymous\"></script>\n<script type=\"application/javascript\">define('jquery', [],function() {return window.jQuery;})</script>\n"
+        "<script src=\"https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js\" integrity=\"sha512-c3Nl8+7g4LMSTdrm621y7kf9v3SDPnhxLNhcjFJbKECVnmZHTdo+IRO05sNLTH/D3vA6u1X32ehoLC7WFVdheg==\" crossorigin=\"anonymous\"></script>\n<script src=\"https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js\" integrity=\"sha512-bLT0Qm9VnAYZDflyKcBaQ2gg0hSYNQrJ8RilYldYQ1FxQYoCLtUjuuRuZo+fjqhx/qtq/1itJ0C2ejDxltZVFg==\" crossorigin=\"anonymous\" data-relocate-top=\"true\"></script>\n<script type=\"application/javascript\">define('jquery', [],function() {return window.jQuery;})</script>\n"
       ]
     }
   }
diff --git a/docs/_quarto.yml b/docs/_quarto.yml
index b3c11e449..3a527c34f 100644
--- a/docs/_quarto.yml
+++ b/docs/_quarto.yml
@@ -45,8 +45,8 @@ website:
     type: overlay
     algolia:
       index-name: prod_ibis
-      application-id: HS77W8GWM1
-      search-only-api-key: 8ca4fcd24da322db857322ae4f79d6f3
+      application-id: TNU9HG3L41
+      search-only-api-key: 26350948d1961209df16d9717a7e01d6
       analytics-events: true
       show-logo: true
 
@@ -298,10 +298,6 @@ quartodoc:
             - name: param
               dynamic: true
               signature_name: full
-            - name: NA
-              # Ideally exposed under `ibis` but that doesn't seem to work??
-              package: ibis.expr.api
-              signature_name: full
             - name: "null"
               dynamic: true
               signature_name: full
diff --git a/docs/posts/campaign-finance/index.qmd b/docs/posts/campaign-finance/index.qmd
index 3d8d9fc19..a2a0a287e 100644
--- a/docs/posts/campaign-finance/index.qmd
+++ b/docs/posts/campaign-finance/index.qmd
@@ -245,7 +245,7 @@ def get_election_type(pgi: StringValue) -> StringValue:
         "E": "recount",
     }
     first_letter = pgi[0]
-    return first_letter.substitute(election_types, else_=ibis.NA)
+    return first_letter.substitute(election_types, else_=ibis.null())
 
 
 cleaned = cleaned.mutate(election_type=get_election_type(_.TRANSACTION_PGI)).drop(
diff --git a/docs/posts/ibis-duckdb-geospatial/index.qmd b/docs/posts/ibis-duckdb-geospatial/index.qmd
index 6271eb7b4..debf9df5f 100644
--- a/docs/posts/ibis-duckdb-geospatial/index.qmd
+++ b/docs/posts/ibis-duckdb-geospatial/index.qmd
@@ -164,20 +164,29 @@ streets_gdf.crs = "EPSG:26918"
 ```
 
 ```{python}
-import leafmap.deckgl as leafmap  # <1>
+from lonboard import Map, ScatterplotLayer, PathLayer, PolygonLayer
 ```
 
-1. `leafmap.deckgl` allows us to visualize multiple layers
-
 ```{python}
-m = leafmap.Map()
-
-m.add_vector(broad_station_gdf, get_fill_color="blue")
-m.add_vector(sts_near_broad_gdf, get_color="red", opacity=0.5)
-m.add_vector(streets_gdf, get_color="grey", zoom_to_layer=False, opacity=0.3)
+broad_station_layer = ScatterplotLayer.from_geopandas(
+    broad_station_gdf, get_fill_color="blue", get_radius=5
+)
+sts_near_broad_layer = PathLayer.from_geopandas(
+    sts_near_broad_gdf, get_color="red", opacity=0.4, get_width=2
+)
+streets_layer = PathLayer.from_geopandas(streets_gdf, get_color="grey", opacity=0.3)
+m = Map(
+    [
+        broad_station_layer,
+        sts_near_broad_layer,
+        streets_layer,
+    ],
+    view_state={"longitude": -74.01066, "latitude": 40.7069, "zoom": 16}
+)
 m
 ```
 
+
 You can zoom in and out, and hover over the map to check on the street names.
 
 ### `buffer` (ST_Buffer)
@@ -234,15 +243,38 @@ h_near_broad_gdf.crs = "EPSG:26918"
 
 h_street_gdf = h_street.to_pandas()
 h_street_gdf.crs = "EPSG:26918"
+```
 
 
-mh = leafmap.Map()
-mh.add_vector(broad_station_gdf, get_fill_color="orange")
-mh.add_vector(broad_station_zone, get_fill_color="orange", opacity=0.1)
-mh.add_vector(h_near_broad_gdf, get_fill_color="red", opacity=0.5)
-mh.add_vector(h_street_gdf, get_color="blue", opacity=0.3)
-mh.add_vector(streets_gdf, get_color="grey", zoom_to_layer=False, opacity=0.2)
-
+```{python}
+broad_station_layer = ScatterplotLayer.from_geopandas(
+    broad_station_gdf, get_fill_color="orange", get_radius=5
+)
+
+broad_station_zone_layer = PolygonLayer.from_geopandas(
+    broad_station_zone, get_fill_color="orange", opacity=0.1
+)
+
+h_near_broad_layer = ScatterplotLayer.from_geopandas(
+    h_near_broad_gdf, get_fill_color="red", get_radius=5
+)
+
+h_street_layer = PathLayer.from_geopandas(
+    h_street_gdf, get_color="blue", opacity=0.5, get_width=2
+)
+
+streets_layer = PathLayer.from_geopandas(streets_gdf, get_color="grey", opacity=0.3)
+
+mh = Map(
+    [
+        broad_station_layer,
+        broad_station_zone_layer,
+        h_near_broad_layer,
+        h_street_layer,
+        streets_layer,
+    ],
+    view_state={"longitude": -74.01066, "latitude": 40.7069, "zoom": 16}
+)
 mh
 ```
 
diff --git a/docs/tutorials/ibis-for-pandas-users.qmd b/docs/tutorials/ibis-for-pandas-users.qmd
index 876fe4ac0..e0fc2f590 100644
--- a/docs/tutorials/ibis-for-pandas-users.qmd
+++ b/docs/tutorials/ibis-for-pandas-users.qmd
@@ -507,7 +507,7 @@ represented by `NaN`. This can be confusing when working with numeric data,
 since `NaN` is also a valid floating point value (along with `+/-inf`).
 
 In Ibis, we try to be more precise: All data types are nullable, and we use
-`ibis.NA` to represent `NULL` values, and all datatypes have a `.isnull()` method.
+`ibis.null()` to represent `NULL` values, and all datatypes have a `.isnull()` method.
 For floating point values, we use different values for `NaN` and `+/-inf`, and there
 are the additional methods `.isnan()` and `.isinf()`.
 
diff --git a/docs/tutorials/ibis-for-sql-users.qmd b/docs/tutorials/ibis-for-sql-users.qmd
index 577f7b015..534090bfc 100644
--- a/docs/tutorials/ibis-for-sql-users.qmd
+++ b/docs/tutorials/ibis-for-sql-users.qmd
@@ -522,10 +522,10 @@ ibis.to_sql(expr)
 
 ### Using `NULL` in expressions
 
-To use `NULL` in an expression, either use the special `ibis.NA` value:
+To use `NULL` in an expression, use `ibis.null()` value:
 
 ```{python}
-pos_two = (t.two > 0).ifelse(t.two, ibis.NA)
+pos_two = (t.two > 0).ifelse(t.two, ibis.null())
 expr = t.mutate(two_positive=pos_two)
 ibis.to_sql(expr)
 ```
diff --git a/ibis/__init__.py b/ibis/__init__.py
index 2ec14e182..28924feb2 100644
--- a/ibis/__init__.py
+++ b/ibis/__init__.py
@@ -4,6 +4,9 @@ from __future__ import annotations
 
 __version__ = "9.0.0"
 
+import warnings
+from typing import Any
+
 from ibis import examples, util
 from ibis.backends import BaseBackend
 from ibis.common.exceptions import IbisError
@@ -36,7 +39,7 @@ def __dir__() -> list[str]:
     return sorted(out)
 
 
-def __getattr__(name: str) -> BaseBackend:
+def load_backend(name: str) -> BaseBackend:
     """Load backends in a lazy way with `ibis.<backend-name>`.
 
     This also registers the backend options.
@@ -52,6 +55,7 @@ def __getattr__(name: str) -> BaseBackend:
     attribute is "cached", so this function is only called the first time.
 
     """
+
     entry_points = {ep for ep in util.backend_entry_points() if ep.name == name}
 
     if not entry_points:
@@ -95,7 +99,6 @@ def __getattr__(name: str) -> BaseBackend:
     # - connect
     # - compile
     # - has_operation
-    # - add_operation
     # - _from_url
     # - _to_sqlglot
     #
@@ -116,7 +119,6 @@ def __getattr__(name: str) -> BaseBackend:
     proxy.connect = connect
     proxy.compile = backend.compile
     proxy.has_operation = backend.has_operation
-    proxy.add_operation = backend.add_operation
     proxy.name = name
     proxy._from_url = backend._from_url
     proxy._to_sqlglot = backend._to_sqlglot
@@ -125,3 +127,17 @@ def __getattr__(name: str) -> BaseBackend:
         setattr(proxy, name, getattr(backend, name))
 
     return proxy
+
+
+def __getattr__(name: str) -> Any:
+    if name == "NA":
+        warnings.warn(
+            "The 'ibis.NA' constant is deprecated as of v9.1 and will be removed in a future "
+            "version. Use 'ibis.null()' instead.",
+            DeprecationWarning,
+            stacklevel=2,
+        )
+
+        return null()  # noqa: F405
+    else:
+        return load_backend(name)
diff --git a/ibis/backends/__init__.py b/ibis/backends/__init__.py
index 2a434f19e..f2341b2f6 100644
--- a/ibis/backends/__init__.py
+++ b/ibis/backends/__init__.py
@@ -20,7 +20,7 @@ from ibis import util
 from ibis.common.caching import RefCountedCache
 
 if TYPE_CHECKING:
-    from collections.abc import Callable, Iterable, Iterator, Mapping, MutableMapping
+    from collections.abc import Iterable, Iterator, Mapping, MutableMapping
 
     import pandas as pd
     import polars as pl
@@ -1051,23 +1051,6 @@ class BaseBackend(abc.ABC, _FileIOHandler):
     def execute(self, expr: ir.Expr) -> Any:
         """Execute an expression."""
 
-    def add_operation(self, operation: ops.Node) -> Callable:
-        """Add a translation function to the backend for a specific operation.
-
-        Operations are defined in `ibis.expr.operations`, and a translation
-        function receives the translator object and an expression as
-        parameters, and returns a value depending on the backend.
-        """
-        if not hasattr(self, "compiler"):
-            raise RuntimeError("Only SQL-based backends support `add_operation`")
-
-        def decorator(translation_function: Callable) -> None:
-            self.compiler.translator_class.add_operation(
-                operation, translation_function
-            )
-
-        return decorator
-
     @abc.abstractmethod
     def create_table(
         self,
diff --git a/ibis/expr/api.py b/ibis/expr/api.py
index 9eb8039bf..e99f4d8fa 100644
--- a/ibis/expr/api.py
+++ b/ibis/expr/api.py
@@ -57,7 +57,6 @@ __all__ = (
     "Column",
     "Deferred",
     "Expr",
-    "NA",
     "Scalar",
     "Schema",
     "Table",
@@ -197,35 +196,6 @@ e = ops.E().to_expr()
 pi = ops.Pi().to_expr()
 
 
-NA = null()
-"""The NULL scalar.
-
-This is an untyped NULL. If you want a typed NULL, use eg `ibis.null(str)`.
-
-Examples
---------
->>> import ibis
->>> ibis.options.interactive = True
->>> ibis.NA.isnull()
-┌──────┐
-│ True │
-└──────┘
-
-datatype-specific methods aren't available on `NA`:
-
->>> ibis.NA.upper()  # quartodoc: +EXPECTED_FAILURE
-Traceback (most recent call last):
-  ...
-AttributeError: 'NullScalar' object has no attribute 'upper'
-
-Instead, use the typed `ibis.null`:
-
->>> ibis.null(str).upper().isnull()
-┌──────┐
-│ True │
-└──────┘
-"""
-
 deferred = _
 """Deferred expression object.
 
diff --git a/poetry.lock b/poetry.lock
index 8b495d65d..006a902e6 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
 
 [[package]]
 name = "aiohttp"
@@ -170,13 +170,13 @@ trio = ["trio (>=0.23)"]
 
 [[package]]
 name = "anywidget"
-version = "0.7.1"
+version = "0.9.11"
 description = "custom jupyter widgets made easy"
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "anywidget-0.7.1-py3-none-any.whl", hash = "sha256:17828c6e255a7066c960636cbda045d55b08c996a2fc4e886e4c8507db6e9407"},
-    {file = "anywidget-0.7.1.tar.gz", hash = "sha256:1c4cb51063f8ec5172b8c8efb1bfb79861d340c3491220d56b8c2a6aa8db3b3d"},
+    {file = "anywidget-0.9.11-py3-none-any.whl", hash = "sha256:b5dfaed080cab0fd6bca0f444bae0717336d8d55cee09713856ec44b39bf2c7a"},
+    {file = "anywidget-0.9.11.tar.gz", hash = "sha256:58dded8e86ec27ce28760c9457317663113d9af608947ee6a8d0ecb528b22cdc"},
 ]
 
 [package.dependencies]
@@ -186,7 +186,7 @@ typing-extensions = ">=4.2.0"
 
 [package.extras]
 dev = ["comm (>=0.1.0)", "watchfiles (>=0.18.0)"]
-test = ["black[jupyter]", "ipython (<8.13)", "msgspec", "pydantic", "pytest", "pytest-cov", "ruff"]
+test = ["ipython (<8.13)", "msgspec", "mypy (==1.10.0)", "pydantic", "pytest", "pytest-cov", "ruff"]
 
 [[package]]
 name = "appdirs"
@@ -629,38 +629,6 @@ webencodings = "*"
 [package.extras]
 css = ["tinycss2 (>=1.1.0,<1.3)"]
 
-[[package]]
-name = "bqplot"
-version = "0.12.43"
-description = "Interactive plotting for the Jupyter notebook, using d3.js and ipywidgets."
-optional = false
-python-versions = ">=3.6"
-files = [
-    {file = "bqplot-0.12.43-py2.py3-none-any.whl", hash = "sha256:65aa2f10bf2e4b500e628d5c1b876e6d2cdc2b82883200e4b1bb65edc17d111b"},
-    {file = "bqplot-0.12.43.tar.gz", hash = "sha256:f2b469d1f03df4f51873cb0406a24b56cdf4a835fc5aa9560776fc636b8b3449"},
-]
-
-[package.dependencies]
-ipywidgets = ">=7.5.0,<9"
-numpy = ">=1.10.4"
-pandas = ">=1.0.0,<3.0.0"
-traitlets = ">=4.3.0"
-traittypes = ">=0.0.6"
-
-[[package]]
-name = "branca"
-version = "0.7.2"
-description = "Generate complex HTML+JS pages with Python"
-optional = false
-python-versions = ">=3.7"
-files = [
-    {file = "branca-0.7.2-py3-none-any.whl", hash = "sha256:853a359c34d08fd06498be762d8be9932750db4049cac11e25dd6f23562e25c2"},
-    {file = "branca-0.7.2.tar.gz", hash = "sha256:ca4c94643ef31b819987ca5bd19c6009ea17b440baa3aac04628545f7a4da023"},
-]
-
-[package.dependencies]
-jinja2 = ">=3"
-
 [[package]]
 name = "build"
 version = "1.2.1"
@@ -1113,20 +1081,6 @@ files = [
     {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"},
 ]
 
-[[package]]
-name = "colour"
-version = "0.1.5"
-description = "converts and manipulates various color representation (HSL, RVB, web, X11, ...)"
-optional = false
-python-versions = "*"
-files = [
-    {file = "colour-0.1.5-py2.py3-none-any.whl", hash = "sha256:33f6db9d564fadc16e59921a56999b79571160ce09916303d35346dddc17978c"},
-    {file = "colour-0.1.5.tar.gz", hash = "sha256:af20120fefd2afede8b001fbef2ea9da70ad7d49fafdb6489025dae8745c3aee"},
-]
-
-[package.extras]
-test = ["nose"]
-
 [[package]]
 name = "comm"
 version = "0.2.2"
@@ -1529,7 +1483,7 @@ toml = ["tomli"]
 name = "duckdb"
 version = "1.0.0"
 description = "DuckDB in-process database"
-optional = false
+optional = true
 python-versions = ">=3.7.0"
 files = [
     {file = "duckdb-1.0.0-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:4a8ce2d1f9e1c23b9bab3ae4ca7997e9822e21563ff8f646992663f66d050211"},
@@ -1800,27 +1754,6 @@ calc = ["shapely"]
 s3 = ["boto3 (>=1.3.1)"]
 test = ["fiona[s3]", "pytest (>=7)", "pytest-cov", "pytz"]
 
-[[package]]
-name = "folium"
-version = "0.16.0"
-description = "Make beautiful maps with Leaflet.js & Python"
-optional = false
-python-versions = ">=3.7"
-files = [
-    {file = "folium-0.16.0-py2.py3-none-any.whl", hash = "sha256:ba72505db18bef995c880da19457d2b10c931db8059af5f6ccec9310d262b584"},
-    {file = "folium-0.16.0.tar.gz", hash = "sha256:2585ee9253dc758d3a365534caa6fb5fa0c244646db4dc5819afc67bbd4daabb"},
-]
-
-[package.dependencies]
-branca = ">=0.6.0"
-jinja2 = ">=2.9"
-numpy = "*"
-requests = "*"
-xyzservices = "*"
-
-[package.extras]
-testing = ["pytest"]
-
 [[package]]
 name = "fonttools"
 version = "4.53.0"
@@ -2046,37 +1979,6 @@ requests = "*"
 crc = ["crcmod"]
 gcsfuse = ["fusepy"]
 
-[[package]]
-name = "gdown"
-version = "5.2.0"
-description = "Google Drive Public File/Folder Downloader"
-optional = false
-python-versions = ">=3.8"
-files = [
-    {file = "gdown-5.2.0-py3-none-any.whl", hash = "sha256:33083832d82b1101bdd0e9df3edd0fbc0e1c5f14c9d8c38d2a35bf1683b526d6"},
-    {file = "gdown-5.2.0.tar.gz", hash = "sha256:2145165062d85520a3cd98b356c9ed522c5e7984d408535409fd46f94defc787"},
-]
-
-[package.dependencies]
-beautifulsoup4 = "*"
-filelock = "*"
-requests = {version = "*", extras = ["socks"]}
-tqdm = "*"
-
-[package.extras]
-test = ["build", "mypy", "pytest", "pytest-xdist", "ruff", "twine", "types-requests", "types-setuptools"]
-
-[[package]]
-name = "geojson"
-version = "3.1.0"
-description = "Python bindings and utilities for GeoJSON"
-optional = false
-python-versions = ">=3.7"
-files = [
-    {file = "geojson-3.1.0-py3-none-any.whl", hash = "sha256:68a9771827237adb8c0c71f8527509c8f5bef61733aa434cefc9c9d4f0ebe8f3"},
-    {file = "geojson-3.1.0.tar.gz", hash = "sha256:58a7fa40727ea058efc28b0e9ff0099eadf6d0965e04690830208d3ef571adac"},
-]
-
 [[package]]
 name = "geopandas"
 version = "0.14.4"
@@ -2758,38 +2660,6 @@ files = [
     {file = "installer-0.7.0.tar.gz", hash = "sha256:a26d3e3116289bb08216e0d0f7d925fcef0b0194eedfa0c944bcaaa106c4b631"},
 ]
 
-[[package]]
-name = "ipyevents"
-version = "2.0.2"
-description = "A custom widget for returning mouse and keyboard events to Python"
-optional = false
-python-versions = ">=3.6"
-files = [
-    {file = "ipyevents-2.0.2-py3-none-any.whl", hash = "sha256:60c2a9e992bdc41e8577aa27e57b124efafa48a59a3bff886029fe5700d546b3"},
-    {file = "ipyevents-2.0.2.tar.gz", hash = "sha256:26e878b0c5854bc8b6bd6a2bd2c89b314ebe86fda642f4d2434051545bab258f"},
-]
-
-[package.dependencies]
-ipywidgets = ">=7.6.0"
-
-[package.extras]
-docs = ["jupyterlab (>=3)", "nbsphinx", "sphinx"]
-test = ["nbval", "pytest", "pytest-cov"]
-
-[[package]]
-name = "ipyfilechooser"
-version = "0.6.0"
-description = "Python file chooser widget for use in Jupyter/IPython in conjunction with ipywidgets"
-optional = false
-python-versions = "*"
-files = [
-    {file = "ipyfilechooser-0.6.0-py3-none-any.whl", hash = "sha256:4555c24b30b819c91dc0ae5e6f7e4cf8f90e5cca531a9209a1fe4deee288d5c5"},
-    {file = "ipyfilechooser-0.6.0.tar.gz", hash = "sha256:41df9e4395a924f8e1b78e2804dbe5066dc3fdc233fb07fecfcdc2a0c9a7d8d3"},
-]
-
-[package.dependencies]
-ipywidgets = "*"
-
 [[package]]
 name = "ipykernel"
 version = "6.29.4"
@@ -2823,24 +2693,6 @@ pyqt5 = ["pyqt5"]
 pyside6 = ["pyside6"]
 test = ["flaky", "ipyparallel", "pre-commit", "pytest (>=7.0)", "pytest-asyncio (>=0.23.5)", "pytest-cov", "pytest-timeout"]
 
-[[package]]
-name = "ipyleaflet"
-version = "0.19.1"
-description = "A Jupyter widget for dynamic Leaflet maps"
-optional = false
-python-versions = ">=3.8"
-files = [
-    {file = "ipyleaflet-0.19.1-py3-none-any.whl", hash = "sha256:ffb90f67576c22c85438490bb32af1f30de88993bfb5dc167503e3a278252139"},
-    {file = "ipyleaflet-0.19.1.tar.gz", hash = "sha256:3454dbc0d360150516ea2a3ab4109503ceb9b7347fa1746a6cafd889075a0ff8"},
-]
-
-[package.dependencies]
-branca = ">=0.5.0"
-ipywidgets = ">=7.6.0,<9"
-jupyter-leaflet = ">=0.19,<0.20"
-traittypes = ">=0.2.1,<3"
-xyzservices = ">=2021.8.1"
-
 [[package]]
 name = "ipython"
 version = "8.25.0"
@@ -2879,20 +2731,6 @@ qtconsole = ["qtconsole"]
 test = ["pickleshare", "pytest", "pytest-asyncio (<0.22)", "testpath"]
 test-extra = ["curio", "ipython[test]", "matplotlib (!=3.2.0)", "nbformat", "numpy (>=1.23)", "pandas", "trio"]
 
-[[package]]
-name = "ipytree"
-version = "0.2.2"
-description = "A Tree Widget using jsTree"
-optional = false
-python-versions = "*"
-files = [
-    {file = "ipytree-0.2.2-py2.py3-none-any.whl", hash = "sha256:744dc1a02c3ec26df8a5ecd87d085a67dc8232a1def6048834403ddcf3b64143"},
-    {file = "ipytree-0.2.2.tar.gz", hash = "sha256:d53d739bbaaa45415733cd06e0dc420a2af3d173438617db472a517bc7a61e56"},
-]
-
-[package.dependencies]
-ipywidgets = ">=7.5.0,<9"
-
 [[package]]
 name = "ipywidgets"
 version = "8.1.3"
@@ -3190,17 +3028,6 @@ cli = ["click", "rich"]
 docs = ["jupyterlite-sphinx", "myst-parser", "pydata-sphinx-theme", "sphinxcontrib-spelling"]
 test = ["click", "pre-commit", "pytest (>=7.0)", "pytest-asyncio (>=0.19.0)", "pytest-console-scripts", "rich"]
 
-[[package]]
-name = "jupyter-leaflet"
-version = "0.19.1"
-description = "ipyleaflet extensions for JupyterLab and Jupyter Notebook"
-optional = false
-python-versions = ">=3.8"
-files = [
-    {file = "jupyter_leaflet-0.19.1-py3-none-any.whl", hash = "sha256:8001a7304e9262394b8f896003539438467bed712bb9330dd65785bd9a5f8add"},
-    {file = "jupyter_leaflet-0.19.1.tar.gz", hash = "sha256:f4c1ab7a8b2c91d01a0940d1bab6543c57e91dca6425789632780eef7d58b266"},
-]
-
 [[package]]
 name = "jupyter-lsp"
 version = "2.2.5"
@@ -3543,50 +3370,6 @@ files = [
     {file = "kiwisolver-1.4.5.tar.gz", hash = "sha256:e57e563a57fb22a142da34f38acc2fc1a5c864bc29ca1517a88abc963e60d6ec"},
 ]
 
-[[package]]
-name = "leafmap"
-version = "0.30.1"
-description = "A Python package for geospatial analysis and interactive mapping in a Jupyter environment."
-optional = false
-python-versions = ">=3.8"
-files = [
-    {file = "leafmap-0.30.1-py2.py3-none-any.whl", hash = "sha256:18223679b47d436cebde1d51aa96061de5f033f0da6eb27681325dafdda74eb5"},
-    {file = "leafmap-0.30.1.tar.gz", hash = "sha256:980caf9ebbb45c2ead56a73531c5204824154d349186e6e8fec5ae69d4035d9c"},
-]
-
-[package.dependencies]
-bqplot = "*"
-colour = "*"
-duckdb = "*"
-folium = "*"
-gdown = "*"
-geojson = "*"
-ipyevents = "*"
-ipyfilechooser = "*"
-ipyleaflet = "*"
-ipywidgets = "*"
-matplotlib = "*"
-numpy = "*"
-pandas = "*"
-plotly = "*"
-pyshp = "*"
-pystac-client = "*"
-python-box = "*"
-scooby = "*"
-whiteboxgui = "*"
-xyzservices = "*"
-
-[package.extras]
-ai = ["geopandas", "localtileserver", "osmnx", "pytorch-lightning", "rastervision"]
-all = ["PyYAML (==6.0.1)", "black", "black[jupyter]", "bokeh", "boto3", "codespell", "cogeo-mosaic", "deadlink", "earthaccess", "ffmpeg-python", "geopandas", "googledrivedownloader", "gradio", "h5py", "ipygany", "ipysheet", "ipyvtklink", "jupyter-bokeh", "jupyterlab (>=3.0.0)", "keplergl", "laspy", "localtileserver", "lonboard", "mapclassify (>=2.4.0)", "mss", "netcdf4", "osmnx", "owslib", "palettable", "panel", "plotly", "pmtiles", "psycopg2", "pycrs", "pydeck", "pyntcloud[las]", "pyvista-xarray", "rasterio", "rasterstats", "rio-cogeo", "rioxarray", "sqlalchemy", "streamlit-folium", "xarray-leaflet"]
-apps = ["solara", "streamlit-folium", "voila"]
-backends = ["bokeh", "keplergl", "plotly", "pydeck"]
-lidar = ["ipygany", "ipyvtklink", "laspy", "panel", "pyntcloud[las]", "pyvista"]
-pmtiles = ["flask", "flask-cors", "pmtiles"]
-raster = ["jupyter-server-proxy", "localtileserver", "netcdf4", "py3dep", "pynhd", "rio-cogeo", "rioxarray"]
-sql = ["psycopg2", "sqlalchemy"]
-vector = ["flask", "flask-cors", "geopandas", "lonboard", "mapclassify", "osmnx", "pmtiles"]
-
 [[package]]
 name = "locket"
 version = "1.0.0"
@@ -3600,23 +3383,27 @@ files = [
 
 [[package]]
 name = "lonboard"
-version = "0.4.0"
+version = "0.9.1"
 description = "Python library for fast, interactive geospatial vector data visualization in Jupyter."
 optional = false
-python-versions = ">=3.8,<4.0"
+python-versions = "<4.0,>=3.8"
 files = [
-    {file = "lonboard-0.4.0-py3-none-any.whl", hash = "sha256:2c521805c05db4c8efdbd7cb19174d3f5e1c43723b7c6db358b2f158d96e671f"},
-    {file = "lonboard-0.4.0.tar.gz", hash = "sha256:c60ccaa5ac06ad2fdad88e146cc2206389d59e0dd4d53bd6de8bab26cae81e7c"},
+    {file = "lonboard-0.9.1-py3-none-any.whl", hash = "sha256:df2a9cee9b1210520e0b705e310f87cc60f4f7f2df5a880ea437aac9b3f2fa64"},
+    {file = "lonboard-0.9.1.tar.gz", hash = "sha256:e3b6c0056b936fcf9c0952086997812c1d5235ba4551fa4f984ee20beab9c94f"},
 ]
 
 [package.dependencies]
-anywidget = ">=0.7.1,<0.8.0"
+anywidget = ">=0.9.0,<0.10.0"
 geopandas = ">=0.13"
 matplotlib = ">=3.7,<4.0"
 palettable = ">=3.3.3,<4.0.0"
 pandas = ">=2,<3"
 pyarrow = ">=14.0.1"
 shapely = ">=2,<3"
+typing-extensions = {version = ">=4.6.0,<5.0.0", markers = "python_version < \"3.12\""}
+
+[package.extras]
+cli = ["click (>=8.1.7,<9.0.0)", "pyogrio (>=0.7.2,<0.8.0)"]
 
 [[package]]
 name = "lz4"
@@ -3949,6 +3736,7 @@ files = [
     {file = "msgpack-1.0.8-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:5fbb160554e319f7b22ecf530a80a3ff496d38e8e07ae763b9e82fadfe96f273"},
     {file = "msgpack-1.0.8-cp39-cp39-win32.whl", hash = "sha256:f9af38a89b6a5c04b7d18c492c8ccf2aee7048aff1ce8437c4683bb5a1df893d"},
     {file = "msgpack-1.0.8-cp39-cp39-win_amd64.whl", hash = "sha256:ed59dd52075f8fc91da6053b12e8c89e37aa043f8986efd89e61fae69dc1b011"},
+    {file = "msgpack-1.0.8-py3-none-any.whl", hash = "sha256:24f727df1e20b9876fa6e95f840a2a2651e34c0ad147676356f4bf5fbb0206ca"},
     {file = "msgpack-1.0.8.tar.gz", hash = "sha256:95c02b0e27e706e48d0e5426d1710ca78e0f0628d6e89d5b5a5b91a5f12274f3"},
 ]
 
@@ -5602,29 +5390,6 @@ files = [
     {file = "pyproject_hooks-1.1.0.tar.gz", hash = "sha256:4b37730834edbd6bd37f26ece6b44802fb1c1ee2ece0e54ddff8bfc06db86965"},
 ]
 
-[[package]]
-name = "pyshp"
-version = "2.3.1"
-description = "Pure Python read/write support for ESRI Shapefile format"
-optional = false
-python-versions = ">=2.7"
-files = [
-    {file = "pyshp-2.3.1-py2.py3-none-any.whl", hash = "sha256:67024c0ccdc352ba5db777c4e968483782dfa78f8e200672a90d2d30fd8b7b49"},
-    {file = "pyshp-2.3.1.tar.gz", hash = "sha256:4caec82fd8dd096feba8217858068bacb2a3b5950f43c048c6dc32a3489d5af1"},
-]
-
-[[package]]
-name = "pysocks"
-version = "1.7.1"
-description = "A Python SOCKS client module. See https://github.com/Anorov/PySocks for more information."
-optional = false
-python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
-files = [
-    {file = "PySocks-1.7.1-py27-none-any.whl", hash = "sha256:08e69f092cc6dbe92a0fdd16eeb9b9ffbc13cadfe5ca4c7bd92ffb078b293299"},
-    {file = "PySocks-1.7.1-py3-none-any.whl", hash = "sha256:2725bd0a9925919b9b51739eea5f9e2bae91e83288108a9ad338b2e3a4435ee5"},
-    {file = "PySocks-1.7.1.tar.gz", hash = "sha256:3f8804571ebe159c380ac6de37643bb4685970655d3bba243530d6558b799aa0"},
-]
-
 [[package]]
 name = "pyspark"
 version = "3.5.1"
@@ -5645,49 +5410,6 @@ mllib = ["numpy (>=1.15)"]
 pandas-on-spark = ["numpy (>=1.15)", "pandas (>=1.0.5)", "pyarrow (>=4.0.0)"]
 sql = ["numpy (>=1.15)", "pandas (>=1.0.5)", "pyarrow (>=4.0.0)"]
 
-[[package]]
-name = "pystac"
-version = "1.10.1"
-description = "Python library for working with the SpatioTemporal Asset Catalog (STAC) specification"
-optional = false
-python-versions = ">=3.9"
-files = [
-    {file = "pystac-1.10.1-py3-none-any.whl", hash = "sha256:a7c31b3dacc44dfc955d9da8c7351c7b5b99100254b36301a1e312709b51bf2f"},
-    {file = "pystac-1.10.1.tar.gz", hash = "sha256:4617fe5315a79785f79b616b8ac248ba3d4d561457c8300b34573309715808cd"},
-]
-
-[package.dependencies]
-jsonschema = {version = ">=4.18,<5.0", optional = true, markers = "extra == \"validation\""}
-python-dateutil = ">=2.7.0"
-
-[package.extras]
-bench = ["asv (>=0.6.0,<0.7.0)", "packaging (>=24.0,<25.0)", "virtualenv (>=20.22,<21.0)"]
-docs = ["Sphinx (>=6.2,<7.0)", "boto3 (>=1.28,<2.0)", "ipython (>=8.12,<9.0)", "jinja2 (<4.0)", "jupyter (>=1.0,<2.0)", "nbsphinx (>=0.9.0,<0.10.0)", "pydata-sphinx-theme (>=0.13,<1.0)", "rasterio (>=1.3,<2.0)", "shapely (>=2.0,<3.0)", "sphinx-autobuild (==2024.2.4)", "sphinx-design (>=0.5.0,<0.6.0)", "sphinxcontrib-fulltoc (>=1.2,<2.0)"]
-jinja2 = ["jinja2 (<4.0)"]
-orjson = ["orjson (>=3.5)"]
-test = ["black (>=24.0,<25.0)", "codespell (>=2.2,<3.0)", "coverage (>=7.2,<8.0)", "doc8 (>=1.1,<2.0)", "html5lib (>=1.1,<2.0)", "jinja2 (<4.0)", "jsonschema (>=4.18,<5.0)", "mypy (>=1.2,<2.0)", "orjson (>=3.8,<4.0)", "pre-commit (>=3.2,<4.0)", "pytest (>=8.0,<9.0)", "pytest-cov (>=5.0,<6.0)", "pytest-mock (>=3.10,<4.0)", "pytest-recording (>=0.13.0,<0.14.0)", "requests-mock (>=1.11,<2.0)", "ruff (==0.3.4)", "types-html5lib (>=1.1,<2.0)", "types-jsonschema (>=4.18,<5.0)", "types-orjson (>=3.6,<4.0)", "types-python-dateutil (>=2.8,<3.0)", "types-urllib3 (>=1.26,<2.0)"]
-urllib3 = ["urllib3 (>=1.26)"]
-validation = ["jsonschema (>=4.18,<5.0)"]
-
-[[package]]
-name = "pystac-client"
-version = "0.8.2"
-description = "Python library for working with SpatioTemporal Asset Catalog (STAC) APIs."
-optional = false
-python-versions = ">=3.9"
-files = [
-    {file = "pystac_client-0.8.2-py3-none-any.whl", hash = "sha256:9cb85e10107f5a971e2ac7fc05d046b7a2a3f31dc68bf898024d4bcaec2b01db"},
-]
-
-[package.dependencies]
-pystac = {version = ">=1.10.0", extras = ["validation"]}
-python-dateutil = ">=2.8.2"
-requests = ">=2.28.2"
-
-[package.extras]
-dev = ["black (>=24.0,<25.0)", "codespell (>=2.3.0,<2.4.0)", "coverage (>=7.2,<8.0)", "doc8 (>=1.1.1,<1.2.0)", "importlib-metadata (>=7.0,<8.0)", "mypy (>=1.2,<2.0)", "orjson (>=3.8,<4.0)", "pre-commit (>=3.2,<4.0)", "pytest (>=8.0,<9.0)", "pytest-benchmark (>=4.0.0,<4.1.0)", "pytest-console-scripts (>=1.4.0,<1.5.0)", "pytest-cov (>=5.0,<6.0)", "pytest-recording (>=0.13,<1.0)", "recommonmark (>=0.7.1,<0.8.0)", "requests-mock (>=1.12,<2.0)", "ruff (==0.4.6)", "tomli (>=2.0,<3.0)", "types-python-dateutil (>=2.8.19,<2.10.0)", "types-requests (>=2.31.0,<2.32.0)", "urllib3 (<2)"]
-docs = ["Sphinx (>=7.3,<8.0)", "boto3 (>=1.26,<2.0)", "cartopy (>=0.21,<1.0)", "geojson (>=3.1.0,<3.2.0)", "geopandas (>=0.14.0,<0.15.0)", "geoviews (>=1.9,<2.0)", "hvplot (>=0.10.0,<0.11.0)", "ipykernel (>=6.22,<7.0)", "ipython (>=8.12,<9.0)", "jinja2 (<4.0)", "matplotlib (>=3.8,<4.0)", "myst-parser (>=3.0,<4.0)", "nbsphinx (>=0.9,<1.0)", "pydata-sphinx-theme (>=0.13,<1.0)", "pygeoif (>=1.0,<2.0)", "scipy (>=1.10,<2.0)", "sphinxcontrib-fulltoc (>=1.2,<2.0)"]
-
 [[package]]
 name = "pytest"
 version = "8.2.2"
@@ -5870,38 +5592,6 @@ psutil = ["psutil (>=3.0)"]
 setproctitle = ["setproctitle"]
 testing = ["filelock"]
 
-[[package]]
-name = "python-box"
-version = "7.1.1"
-description = "Advanced Python dictionaries with dot notation access"
-optional = false
-python-versions = ">=3.8"
-files = [
-    {file = "python-box-7.1.1.tar.gz", hash = "sha256:2a3df244a5a79ac8f8447b5d11b5be0f2747d7b141cb2866060081ae9b53cc50"},
-    {file = "python_box-7.1.1-cp310-cp310-macosx_11_0_x86_64.whl", hash = "sha256:81ed1ec0f0ff2370227fc07277c5baca46d190a4747631bad7eb6ab1630fb7d9"},
-    {file = "python_box-7.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8891735b4148e84d348c6eadd2f127152f751c9603e35d43a1f496183a291ac4"},
-    {file = "python_box-7.1.1-cp310-cp310-win_amd64.whl", hash = "sha256:0036fd47d388deaca8ebd65aea905f88ee6ef91d1d8ce34898b66f1824afbe80"},
-    {file = "python_box-7.1.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:aabf8b9ae5dbc8ba431d8cbe0d4cfe737a25d52d68b0f5f2ff34915c21a2c1db"},
-    {file = "python_box-7.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c046608337e723ae4de3206db5d1e1202ed166da2dfdc70c1f9361e72ace5633"},
-    {file = "python_box-7.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:f9266795e9c233874fb5b34fa994054b4fb0371881678e6ec45aec17fc95feac"},
-    {file = "python_box-7.1.1-cp38-cp38-macosx_11_0_x86_64.whl", hash = "sha256:f76b5b7f0cdc07bfdd4200dc24e6e33189bb2ae322137a2b7110fd41891a3157"},
-    {file = "python_box-7.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4ea13c98e05a3ec0ff26f254986a17290b69b5ade209fad081fd628f8fcfaa08"},
-    {file = "python_box-7.1.1-cp38-cp38-win_amd64.whl", hash = "sha256:1b3f346e332dba16df0b0543d319d9e7ce07d93e5ae152175302894352aa2d28"},
-    {file = "python_box-7.1.1-cp39-cp39-macosx_11_0_x86_64.whl", hash = "sha256:24c4ec0ee0278f66321100aaa9c615413da27a14ff43d376a2a3b4665e1d9494"},
-    {file = "python_box-7.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d95e5eec4fc8f3fc5c9cc7347fc2eb4f9187c853d34c90b1658d1eff96cd4eac"},
-    {file = "python_box-7.1.1-cp39-cp39-win_amd64.whl", hash = "sha256:a0f1333c42e81529b6f68c192050df9d4505b803be7ac47f114036b98707f7cf"},
-    {file = "python_box-7.1.1-py3-none-any.whl", hash = "sha256:63b609555554d7a9d4b6e725f8e78ef1717c67e7d386200e03422ad612338df8"},
-]
-
-[package.extras]
-all = ["msgpack", "ruamel.yaml (>=0.17)", "toml"]
-msgpack = ["msgpack"]
-pyyaml = ["PyYAML"]
-ruamel-yaml = ["ruamel.yaml (>=0.17)"]
-toml = ["toml"]
-tomli = ["tomli", "tomli-w"]
-yaml = ["ruamel.yaml (>=0.17)"]
-
 [[package]]
 name = "python-dateutil"
 version = "2.9.0.post0"
@@ -6012,7 +5702,6 @@ files = [
     {file = "PyYAML-6.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:bf07ee2fef7014951eeb99f56f39c9bb4af143d8aa3c21b1677805985307da34"},
     {file = "PyYAML-6.0.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:855fb52b0dc35af121542a76b9a84f8d1cd886ea97c84703eaa6d88e37a2ad28"},
     {file = "PyYAML-6.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:40df9b996c2b73138957fe23a16a4f0ba614f4c0efce1e9406a184b6d07fa3a9"},
-    {file = "PyYAML-6.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a08c6f0fe150303c1c6b71ebcd7213c2858041a7e01975da3a99aed1e7a378ef"},
     {file = "PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6c22bec3fbe2524cde73d7ada88f6566758a8f7227bfbf93a408a9d86bcc12a0"},
     {file = "PyYAML-6.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8d4e9c88387b0f5c7d5f281e55304de64cf7f9c0021a3525bd3b1c542da3b0e4"},
     {file = "PyYAML-6.0.1-cp312-cp312-win32.whl", hash = "sha256:d483d2cdf104e7c9fa60c544d92981f12ad66a457afae824d146093b8c294c54"},
@@ -6397,7 +6086,6 @@ files = [
 certifi = ">=2017.4.17"
 charset-normalizer = ">=2,<4"
 idna = ">=2.5,<4"
-PySocks = {version = ">=1.5.6,<1.5.7 || >1.5.7", optional = true, markers = "extra == \"socks\""}
 urllib3 = ">=1.21.1,<3"
 
 [package.extras]
@@ -6714,20 +6402,6 @@ dev = ["cython-lint (>=0.12.2)", "doit (>=0.36.0)", "mypy", "pycodestyle", "pyde
 doc = ["jupyterlite-pyodide-kernel", "jupyterlite-sphinx (>=0.12.0)", "jupytext", "matplotlib (>=3.5)", "myst-nb", "numpydoc", "pooch", "pydata-sphinx-theme (>=0.15.2)", "sphinx (>=5.0.0)", "sphinx-design (>=0.4.0)"]
 test = ["array-api-strict", "asv", "gmpy2", "hypothesis (>=6.30)", "mpmath", "pooch", "pytest", "pytest-cov", "pytest-timeout", "pytest-xdist", "scikit-umfpack", "threadpoolctl"]
 
-[[package]]
-name = "scooby"
-version = "0.10.0"
-description = "A Great Dane turned Python environment detective"
-optional = false
-python-versions = ">=3.8"
-files = [
-    {file = "scooby-0.10.0-py3-none-any.whl", hash = "sha256:0a3d7e304f8ebb16f69ff7f6360c345d7f50b45f2ddbf7c3d18a6a0dc2cb03a6"},
-    {file = "scooby-0.10.0.tar.gz", hash = "sha256:7ea33c262c0cc6a33c6eeeb5648df787be4f22660e53c114e5fff1b811a8854f"},
-]
-
-[package.extras]
-cpu = ["mkl", "psutil"]
-
 [[package]]
 name = "seaborn"
 version = "0.13.2"
@@ -7366,23 +7040,6 @@ files = [
 docs = ["myst-parser", "pydata-sphinx-theme", "sphinx"]
 test = ["argcomplete (>=3.0.3)", "mypy (>=1.7.0)", "pre-commit", "pytest (>=7.0,<8.2)", "pytest-mock", "pytest-mypy-testing"]
 
-[[package]]
-name = "traittypes"
-version = "0.2.1"
-description = "Scipy trait types"
-optional = false
-python-versions = "*"
-files = [
-    {file = "traittypes-0.2.1-py2.py3-none-any.whl", hash = "sha256:1340af133810b6eee1a2eb2e988f862b0d12b6c2d16f282aaf3207b782134c2e"},
-    {file = "traittypes-0.2.1.tar.gz", hash = "sha256:be6fa26294733e7489822ded4ae25da5b4824a8a7a0e0c2dccfde596e3489bd6"},
-]
-
-[package.dependencies]
-traitlets = ">=4.2.2"
-
-[package.extras]
-test = ["numpy", "pandas", "pytest", "xarray"]
-
 [[package]]
 name = "trino"
 version = "0.328.0"
@@ -7633,37 +7290,6 @@ MarkupSafe = ">=2.1.1"
 [package.extras]
 watchdog = ["watchdog (>=2.3)"]
 
-[[package]]
-name = "whitebox"
-version = "2.3.4"
-description = "An advanced geospatial data analysis platform"
-optional = false
-python-versions = "*"
-files = [
-    {file = "whitebox-2.3.4-py2.py3-none-any.whl", hash = "sha256:4322c5a3b42f1f04a3c80640253b672242e0aefb6b6dea70752f00a220f6d7ed"},
-    {file = "whitebox-2.3.4.tar.gz", hash = "sha256:3479c8127b1928dd0457b6680b9ee4469d0735b2bb220178c678af303d868ef0"},
-]
-
-[package.dependencies]
-Click = ">=6.0"
-
-[[package]]
-name = "whiteboxgui"
-version = "2.3.0"
-description = "An interactive GUI for whitebox-tools in a Jupyter-based environment"
-optional = false
-python-versions = ">=3.5"
-files = [
-    {file = "whiteboxgui-2.3.0-py2.py3-none-any.whl", hash = "sha256:f226783efaba1af1cd55f98bba743da590bccf4dd6cbba519f6c879993b57631"},
-    {file = "whiteboxgui-2.3.0.tar.gz", hash = "sha256:c59dfccb244bc2d7b9ff77c63a81b77d0e1b4a6fc19476449e450e0da009a754"},
-]
-
-[package.dependencies]
-ipyfilechooser = "*"
-ipytree = "*"
-ipywidgets = "*"
-whitebox = "*"
-
 [[package]]
 name = "widgetsnbextension"
 version = "4.0.11"
@@ -7865,17 +7491,6 @@ files = [
     {file = "xxhash-3.4.1.tar.gz", hash = "sha256:0379d6cf1ff987cd421609a264ce025e74f346e3e145dd106c0cc2e3ec3f99a9"},
 ]
 
-[[package]]
-name = "xyzservices"
-version = "2024.6.0"
-description = "Source of XYZ tiles providers"
-optional = false
-python-versions = ">=3.8"
-files = [
-    {file = "xyzservices-2024.6.0-py3-none-any.whl", hash = "sha256:fecb2508f0f2b71c819aecf5df2c03cef001c56a4b49302e640f3b34710d25e4"},
-    {file = "xyzservices-2024.6.0.tar.gz", hash = "sha256:58c1bdab4257d2551b9ef91cd48571f77b7c4d2bc45bf5e3c05ac97b3a4d7282"},
-]
-
 [[package]]
 name = "yarl"
 version = "1.9.4"
@@ -8085,4 +7700,4 @@ visualization = ["graphviz"]
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.10"
-content-hash = "3613d324a066d1a405963af828ebbcf31f2f9b34584c42e583aa177bbc39b2f2"
+content-hash = "0c0c40f5373ddd48a99af6b3e6df5a0ecea75b063ec555ac384299ba639cd6ad"
diff --git a/pyproject.toml b/pyproject.toml
index dbfbc5618..f45bee5c4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -134,8 +134,7 @@ quartodoc = { version = ">=0.6.1,<1", python = ">=3.10,<3.13" }
 requests = { version = ">=2,<3", python = ">=3.10,<3.13" }
 scikit-learn = { version = ">=1.3,<2", python = ">=3.10,<3.13" }
 seaborn = { version = ">=0.12.2,<1", python = ">=3.10,<3.13" }
-leafmap = { version = ">=0.29.6,<0.31", python = ">=3.10,<3.13" }
-lonboard = { version = "==0.4.0", python = ">=3.10,<3.13" }
+lonboard = { version = "0.9.1", python = ">=3.10,<3.13" }
 jupyter-cache = { version = ">=1.0.0,<2", python = ">=3.10,<3.13" }
 jupyterlab = { version = ">=4.1.6,<5", python = ">=3.10,<3.13" }
 jupyterlab-night = { version = ">=0.4.6,<1", python = ">=3.10,<3.13" }
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 5da00ec68..391e13c9a 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -3,7 +3,7 @@ aiosignal==1.3.1 ; python_version >= "3.10" and python_version < "4.0"
 altair==5.3.0 ; python_version >= "3.10" and python_version < "3.13"
 annotated-types==0.7.0 ; python_version >= "3.10" and python_version < "3.13"
 anyio==4.4.0 ; python_version >= "3.10" and python_version < "3.13"
-anywidget==0.7.1 ; python_version >= "3.10" and python_version < "3.13"
+anywidget==0.9.11 ; python_version >= "3.10" and python_version < "3.13"
 appdirs==1.4.4 ; python_version >= "3.10" and python_version < "4.0"
 appnope==0.1.4 ; python_version >= "3.10" and python_version < "3.13" and platform_system == "Darwin"
 argon2-cffi-bindings==21.2.0 ; python_version >= "3.10" and python_version < "3.13"
@@ -22,8 +22,6 @@ bidict==0.23.1 ; python_version >= "3.10" and python_version < "4.0"
 bitarray==2.9.2 ; python_version >= "3.10" and python_version < "4.0"
 black==24.4.2 ; python_version >= "3.10" and python_version < "4.0"
 bleach==6.1.0 ; python_version >= "3.10" and python_version < "3.13"
-bqplot==0.12.43 ; python_version >= "3.10" and python_version < "3.13"
-branca==0.7.2 ; python_version >= "3.10" and python_version < "3.13"
 build==1.2.1 ; python_version >= "3.10" and python_version < "4.0"
 cachecontrol[filecache]==0.14.0 ; python_version >= "3.10" and python_version < "4.0"
 cachetools==5.3.3 ; python_version >= "3.10" and python_version < "4.0"
@@ -40,7 +38,6 @@ cligj==0.7.2 ; python_version >= "3.10" and python_version < "4"
 cloudpickle==3.0.0 ; python_version >= "3.10" and python_version < "4.0"
 codespell[hard-encoding-detection,toml]==2.3.0 ; python_version >= "3.10" and python_version < "4.0"
 colorama==0.4.6 ; python_version >= "3.10" and python_version < "4.0" and (sys_platform == "win32" or platform_system == "Windows" or os_name == "nt" or python_version < "3.13")
-colour==0.1.5 ; python_version >= "3.10" and python_version < "3.13"
 comm==0.2.2 ; python_version >= "3.10" and python_version < "3.13"
 contourpy==1.2.1 ; python_version >= "3.10" and python_version < "3.13"
 coverage[toml]==7.5.3 ; python_version >= "3.10" and python_version < "4.0"
@@ -65,14 +62,11 @@ executing==2.0.1 ; python_version >= "3.10" and python_version < "4.0"
 fastjsonschema==2.19.1 ; python_version >= "3.10" and python_version < "4.0"
 filelock==3.14.0 ; python_version >= "3.10" and python_version < "4.0"
 fiona==1.9.6 ; python_version >= "3.10" and python_version < "4.0"
-folium==0.16.0 ; python_version >= "3.10" and python_version < "3.13"
 fonttools==4.53.0 ; python_version >= "3.10" and python_version < "3.13"
 fqdn==1.5.1 ; python_version >= "3.10" and python_version < "3.13"
 frozenlist==1.4.1 ; python_version >= "3.10" and python_version < "4.0"
 fsspec==2024.6.0 ; python_version >= "3.10" and python_version < "4.0"
 gcsfs==2024.6.0 ; python_version >= "3.10" and python_version < "4.0"
-gdown==5.2.0 ; python_version >= "3.10" and python_version < "3.13"
-geojson==3.1.0 ; python_version >= "3.10" and python_version < "3.13"
 geopandas==0.14.4 ; python_version >= "3.10" and python_version < "4.0"
 google-api-core==2.19.0 ; python_version >= "3.10" and python_version < "4.0"
 google-api-core[grpc]==2.19.0 ; python_version >= "3.10" and python_version < "4.0"
@@ -102,12 +96,8 @@ importlib-resources==6.4.0 ; python_version >= "3.10" and python_version < "4.0"
 impyla==0.19.0 ; python_version >= "3.10" and python_version < "4.0"
 iniconfig==2.0.0 ; python_version >= "3.10" and python_version < "4.0"
 installer==0.7.0 ; python_version >= "3.10" and python_version < "4.0"
-ipyevents==2.0.2 ; python_version >= "3.10" and python_version < "3.13"
-ipyfilechooser==0.6.0 ; python_version >= "3.10" and python_version < "3.13"
 ipykernel==6.29.4 ; python_version >= "3.10" and python_version < "3.13"
-ipyleaflet==0.19.1 ; python_version >= "3.10" and python_version < "3.13"
 ipython==8.25.0 ; python_version >= "3.10" and python_version < "4.0"
-ipytree==0.2.2 ; python_version >= "3.10" and python_version < "3.13"
 ipywidgets==8.1.3 ; python_version >= "3.10" and python_version < "3.13"
 isoduration==20.11.0 ; python_version >= "3.10" and python_version < "3.13"
 itables==2.1.1 ; python_version >= "3.10" and python_version < "3.13"
@@ -125,7 +115,6 @@ jupyter-cache==1.0.0 ; python_version >= "3.10" and python_version < "3.13"
 jupyter-client==8.6.2 ; python_version >= "3.10" and python_version < "3.13"
 jupyter-core==5.7.2 ; python_version >= "3.10" and python_version < "3.13"
 jupyter-events==0.10.0 ; python_version >= "3.10" and python_version < "3.13"
-jupyter-leaflet==0.19.1 ; python_version >= "3.10" and python_version < "3.13"
 jupyter-lsp==2.2.5 ; python_version >= "3.10" and python_version < "3.13"
 jupyter-server-terminals==0.5.3 ; python_version >= "3.10" and python_version < "3.13"
 jupyter-server==2.14.1 ; python_version >= "3.10" and python_version < "3.13"
@@ -138,9 +127,8 @@ jupyterlite-core==0.3.0 ; python_version >= "3.10" and python_version < "3.13"
 jupyterlite-pyodide-kernel==0.3.2 ; python_version >= "3.10" and python_version < "3.13"
 keyring==24.3.1 ; python_version >= "3.10" and python_version < "4.0"
 kiwisolver==1.4.5 ; python_version >= "3.10" and python_version < "3.13"
-leafmap==0.30.1 ; python_version >= "3.10" and python_version < "3.13"
 locket==1.0.0 ; python_version >= "3.10" and python_version < "4.0"
-lonboard==0.4.0 ; python_version >= "3.10" and python_version < "3.13"
+lonboard==0.9.1 ; python_version >= "3.10" and python_version < "3.13"
 lz4==4.3.3 ; python_version >= "3.10" and python_version < "4.0"
 markdown-it-py==3.0.0 ; python_version >= "3.10" and python_version < "4.0"
 markupsafe==2.1.5 ; python_version >= "3.10" and python_version < "4.0"
@@ -220,11 +208,7 @@ pyopenssl==24.1.0 ; python_version >= "3.10" and python_version < "4.0"
 pyparsing==3.1.2 ; python_version >= "3.10" and python_version < "3.13"
 pyproj==3.6.1 ; python_version >= "3.10" and python_version < "4.0"
 pyproject-hooks==1.1.0 ; python_version >= "3.10" and python_version < "4.0"
-pyshp==2.3.1 ; python_version >= "3.10" and python_version < "3.13"
-pysocks==1.7.1 ; python_version >= "3.10" and python_version < "3.13"
 pyspark==3.5.1 ; python_version >= "3.10" and python_version < "4.0"
-pystac-client==0.8.2 ; python_version >= "3.10" and python_version < "3.13"
-pystac[validation]==1.10.1 ; python_version >= "3.10" and python_version < "3.13"
 pytest-benchmark==4.0.0 ; python_version >= "3.10" and python_version < "4.0"
 pytest-clarity==1.0.1 ; python_version >= "3.10" and python_version < "4.0"
 pytest-cov==5.0.0 ; python_version >= "3.10" and python_version < "4.0"
@@ -236,7 +220,6 @@ pytest-snapshot==0.9.0 ; python_version >= "3.10" and python_version < "4.0"
 pytest-timeout==2.3.1 ; python_version >= "3.10" and python_version < "4.0"
 pytest-xdist==3.6.1 ; python_version >= "3.10" and python_version < "4.0"
 pytest==8.2.2 ; python_version >= "3.10" and python_version < "4.0"
-python-box==7.1.1 ; python_version >= "3.10" and python_version < "3.13"
 python-dateutil==2.9.0.post0 ; python_version >= "3.10" and python_version < "4.0"
 python-json-logger==2.0.7 ; python_version >= "3.10" and python_version < "3.13"
 pytz==2024.1 ; python_version >= "3.10" and python_version < "4.0"
@@ -252,7 +235,6 @@ regex==2024.5.15 ; python_version >= "3.10" and python_version < "4.0"
 requests-oauthlib==2.0.0 ; python_version >= "3.10" and python_version < "4.0"
 requests-toolbelt==1.0.0 ; python_version >= "3.10" and python_version < "4.0"
 requests==2.32.3 ; python_version >= "3.10" and python_version < "4.0"
-requests[socks]==2.32.3 ; python_version >= "3.10" and python_version < "3.13"
 rfc3339-validator==0.1.4 ; python_version >= "3.10" and python_version < "3.13"
 rfc3986-validator==0.1.1 ; python_version >= "3.10" and python_version < "3.13"
 rich==13.7.1 ; python_version >= "3.10" and python_version < "4.0"
@@ -261,7 +243,6 @@ rsa==4.9 ; python_version >= "3.10" and python_version < "4"
 ruff==0.4.8 ; python_version >= "3.10" and python_version < "4.0"
 scikit-learn==1.5.0 ; python_version >= "3.10" and python_version < "3.13"
 scipy==1.13.1 ; python_version >= "3.10" and python_version < "3.13"
-scooby==0.10.0 ; python_version >= "3.10" and python_version < "3.13"
 seaborn==0.13.2 ; python_version >= "3.10" and python_version < "3.13"
 secretstorage==3.3.3 ; python_version >= "3.10" and python_version < "4.0" and sys_platform == "linux"
 send2trash==1.8.3 ; python_version >= "3.10" and python_version < "3.13"
@@ -292,7 +273,6 @@ toolz==0.12.1 ; python_version >= "3.10" and python_version < "4.0"
 tornado==6.4.1 ; python_version >= "3.10" and python_version < "3.13"
 tqdm==4.66.4 ; python_version >= "3.10" and python_version < "4.0"
 traitlets==5.14.3 ; python_version >= "3.10" and python_version < "4.0"
-traittypes==0.2.1 ; python_version >= "3.10" and python_version < "3.13"
 trino==0.328.0 ; python_version >= "3.10" and python_version < "4.0"
 trove-classifiers==2024.5.22 ; python_version >= "3.10" and python_version < "4.0"
 types-python-dateutil==2.9.0.20240316 ; python_version >= "3.10" and python_version < "3.13"
@@ -308,12 +288,9 @@ webcolors==24.6.0 ; python_version >= "3.10" and python_version < "3.13"
 webencodings==0.5.1 ; python_version >= "3.10" and python_version < "3.13"
 websocket-client==1.8.0 ; python_version >= "3.10" and python_version < "4.0"
 werkzeug==3.0.3 ; python_version >= "3.10" and python_version < "4.0"
-whitebox==2.3.4 ; python_version >= "3.10" and python_version < "3.13"
-whiteboxgui==2.3.0 ; python_version >= "3.10" and python_version < "3.13"
 widgetsnbextension==4.0.11 ; python_version >= "3.10" and python_version < "3.13"
 xattr==1.1.0 ; python_version >= "3.10" and python_version < "4.0" and sys_platform == "darwin"
 xxhash==3.4.1 ; python_version >= "3.10" and python_version < "4.0"
-xyzservices==2024.6.0 ; python_version >= "3.10" and python_version < "3.13"
 yarl==1.9.4 ; python_version >= "3.10" and python_version < "4.0"
 zipp==3.19.2 ; python_version >= "3.10" and python_version < "4.0"
 zstandard==0.22.0 ; python_version >= "3.10" and python_version < "4.0"


Test patch:
diff --git a/ibis/backends/clickhouse/tests/test_functions.py b/ibis/backends/clickhouse/tests/test_functions.py
index 1385ba7a3..dfe9d5e0f 100644
--- a/ibis/backends/clickhouse/tests/test_functions.py
+++ b/ibis/backends/clickhouse/tests/test_functions.py
@@ -116,8 +116,8 @@ def test_isnull_notnull(con, expr, expected):
     ("expr", "expected"),
     [
         (ibis.coalesce(5, None, 4), 5),
-        (ibis.coalesce(ibis.NA, 4, ibis.NA), 4),
-        (ibis.coalesce(ibis.NA, ibis.NA, 3.14), 3.14),
+        (ibis.coalesce(ibis.null(), 4, ibis.null()), 4),
+        (ibis.coalesce(ibis.null(), ibis.null(), 3.14), 3.14),
     ],
 )
 def test_coalesce(con, expr, expected):
@@ -127,7 +127,7 @@ def test_coalesce(con, expr, expected):
 @pytest.mark.parametrize(
     ("expr", "expected"),
     [
-        (ibis.NA.fill_null(5), 5),
+        (ibis.null().fill_null(5), 5),
         (L(5).fill_null(10), 5),
         (L(5).nullif(5), None),
         (L(10).nullif(5), 10),
@@ -150,7 +150,7 @@ def test_fill_null_nullif(con, expr, expected):
         (L(datetime(2015, 9, 1, hour=14, minute=48, second=5)), "DateTime"),
         (L(date(2015, 9, 1)), "Date"),
         param(
-            ibis.NA,
+            ibis.null(),
             "Null",
             marks=pytest.mark.xfail(
                 raises=AssertionError,
@@ -418,7 +418,7 @@ def test_numeric_builtins_work(alltypes, df):
 def test_null_column(alltypes):
     t = alltypes
     nrows = t.count().execute()
-    expr = t.mutate(na_column=ibis.NA).na_column
+    expr = t.mutate(na_column=ibis.null()).na_column
     result = expr.execute()
     expected = pd.Series([None] * nrows, name="na_column")
     tm.assert_series_equal(result, expected)
diff --git a/ibis/backends/dask/tests/test_window.py b/ibis/backends/dask/tests/test_window.py
index ef2249dd0..2a6d17c67 100644
--- a/ibis/backends/dask/tests/test_window.py
+++ b/ibis/backends/dask/tests/test_window.py
@@ -20,7 +20,7 @@ def sort_kind():
     return "mergesort"
 
 
-default = pytest.mark.parametrize("default", [ibis.NA, ibis.literal("a")])
+default = pytest.mark.parametrize("default", [ibis.null(), ibis.literal("a")])
 row_offset = pytest.mark.parametrize("row_offset", list(map(ibis.literal, [-1, 1, 0])))
 range_offset = pytest.mark.parametrize(
     "range_offset",
@@ -48,7 +48,7 @@ def test_lead(con, t, df, row_offset, default, row_window):
     expr = t.dup_strings.lead(row_offset, default=default).over(row_window)
     result = expr.execute()
     expected = df.dup_strings.shift(con.execute(-row_offset)).compute()
-    if default is not ibis.NA:
+    if default is not ibis.null():
         expected = expected.fillna(con.execute(default))
     tm.assert_series_equal(result, expected, check_names=False)
 
@@ -59,7 +59,7 @@ def test_lag(con, t, df, row_offset, default, row_window):
     expr = t.dup_strings.lag(row_offset, default=default).over(row_window)
     result = expr.execute()
     expected = df.dup_strings.shift(con.execute(row_offset)).compute()
-    if default is not ibis.NA:
+    if default is not ibis.null():
         expected = expected.fillna(con.execute(default))
     tm.assert_series_equal(result, expected, check_names=False)
 
@@ -78,7 +78,7 @@ def test_lead_delta(con, t, pandas_df, range_offset, default, range_window):
         .reindex(pandas_df.plain_datetimes_naive)
         .reset_index(drop=True)
     )
-    if default is not ibis.NA:
+    if default is not ibis.null():
         expected = expected.fillna(con.execute(default))
     tm.assert_series_equal(result, expected, check_names=False)
 
@@ -98,7 +98,7 @@ def test_lag_delta(t, con, pandas_df, range_offset, default, range_window):
         .reindex(pandas_df.plain_datetimes_naive)
         .reset_index(drop=True)
     )
-    if default is not ibis.NA:
+    if default is not ibis.null():
         expected = expected.fillna(con.execute(default))
     tm.assert_series_equal(result, expected, check_names=False)
 
diff --git a/ibis/backends/impala/tests/test_case_exprs.py b/ibis/backends/impala/tests/test_case_exprs.py
index c24d3671a..a195928b1 100644
--- a/ibis/backends/impala/tests/test_case_exprs.py
+++ b/ibis/backends/impala/tests/test_case_exprs.py
@@ -100,6 +100,6 @@ def test_identical_to(mockcon, snapshot):
 
 
 def test_identical_to_special_case(snapshot):
-    expr = ibis.NA.cast("int64").identical_to(ibis.NA.cast("int64")).name("tmp")
+    expr = ibis.null().cast("int64").identical_to(ibis.null().cast("int64")).name("tmp")
     result = ibis.to_sql(expr, dialect="impala")
     snapshot.assert_match(result, "out.sql")
diff --git a/ibis/backends/impala/tests/test_exprs.py b/ibis/backends/impala/tests/test_exprs.py
index 2919d7946..45a0ac96d 100644
--- a/ibis/backends/impala/tests/test_exprs.py
+++ b/ibis/backends/impala/tests/test_exprs.py
@@ -384,7 +384,7 @@ def test_decimal_timestamp_builtins(con):
         dc * 2,
         dc**2,
         dc.cast("double"),
-        api.ifelse(table.l_discount > 0, dc * table.l_discount, api.NA),
+        api.ifelse(table.l_discount > 0, dc * table.l_discount, api.null()),
         dc.fill_null(0),
         ts < (ibis.now() + ibis.interval(months=3)),
         ts < (ibis.timestamp("2005-01-01") + ibis.interval(months=3)),
@@ -632,10 +632,10 @@ def test_unions_with_ctes(con, alltypes):
 @pytest.mark.parametrize(
     ("left", "right", "expected"),
     [
-        (ibis.NA.cast("int64"), ibis.NA.cast("int64"), True),
+        (ibis.null().cast("int64"), ibis.null().cast("int64"), True),
         (L(1), L(1), True),
-        (ibis.NA.cast("int64"), L(1), False),
-        (L(1), ibis.NA.cast("int64"), False),
+        (ibis.null().cast("int64"), L(1), False),
+        (L(1), ibis.null().cast("int64"), False),
         (L(0), L(1), False),
         (L(1), L(0), False),
     ],
diff --git a/ibis/backends/pandas/tests/test_window.py b/ibis/backends/pandas/tests/test_window.py
index 791f29133..d588120b8 100644
--- a/ibis/backends/pandas/tests/test_window.py
+++ b/ibis/backends/pandas/tests/test_window.py
@@ -20,7 +20,7 @@ def sort_kind():
     return "mergesort"
 
 
-default = pytest.mark.parametrize("default", [ibis.NA, ibis.literal("a")])
+default = pytest.mark.parametrize("default", [ibis.null(), ibis.literal("a")])
 row_offset = pytest.mark.parametrize("row_offset", list(map(ibis.literal, [-1, 1, 0])))
 range_offset = pytest.mark.parametrize(
     "range_offset",
@@ -49,7 +49,7 @@ def test_lead(t, df, row_offset, default, row_window):
     expr = t.dup_strings.lead(row_offset, default=default).over(row_window)
     result = expr.execute()
     expected = df.dup_strings.shift(con.execute(-row_offset))
-    if default is not ibis.NA:
+    if default is not ibis.null():
         expected = expected.fillna(con.execute(default))
     tm.assert_series_equal(result, expected.rename("tmp"))
 
@@ -61,7 +61,7 @@ def test_lag(t, df, row_offset, default, row_window):
     expr = t.dup_strings.lag(row_offset, default=default).over(row_window)
     result = expr.execute()
     expected = df.dup_strings.shift(con.execute(row_offset))
-    if default is not ibis.NA:
+    if default is not ibis.null():
         expected = expected.fillna(con.execute(default))
     tm.assert_series_equal(result, expected.rename("tmp"))
 
@@ -80,7 +80,7 @@ def test_lead_delta(t, df, range_offset, default, range_window):
         .reindex(df.plain_datetimes_naive)
         .reset_index(drop=True)
     )
-    if default is not ibis.NA:
+    if default is not ibis.null():
         expected = expected.fillna(con.execute(default))
     tm.assert_series_equal(result, expected.rename("tmp"))
 
@@ -100,7 +100,7 @@ def test_lag_delta(t, df, range_offset, default, range_window):
         .reindex(df.plain_datetimes_naive)
         .reset_index(drop=True)
     )
-    if default is not ibis.NA:
+    if default is not ibis.null():
         expected = expected.fillna(con.execute(default))
     tm.assert_series_equal(result, expected.rename("tmp"))
 
diff --git a/ibis/backends/postgres/tests/test_functions.py b/ibis/backends/postgres/tests/test_functions.py
index 852431d3e..93467491a 100644
--- a/ibis/backends/postgres/tests/test_functions.py
+++ b/ibis/backends/postgres/tests/test_functions.py
@@ -150,7 +150,7 @@ def test_strftime(con, pattern):
     [
         param(L("foo_bar"), "text", id="text"),
         param(L(5), "integer", id="integer"),
-        param(ibis.NA, "null", id="null"),
+        param(ibis.null(), "null", id="null"),
         # TODO(phillipc): should this really be double?
         param(L(1.2345), "numeric", id="numeric"),
         param(
@@ -335,7 +335,7 @@ def test_regexp_extract(con, expr, expected):
 @pytest.mark.parametrize(
     ("expr", "expected"),
     [
-        param(ibis.NA.fill_null(5), 5, id="filled"),
+        param(ibis.null().fill_null(5), 5, id="filled"),
         param(L(5).fill_null(10), 5, id="not_filled"),
         param(L(5).nullif(5), None, id="nullif_null"),
         param(L(10).nullif(5), 10, id="nullif_not_null"),
@@ -349,8 +349,8 @@ def test_fill_null_nullif(con, expr, expected):
     ("expr", "expected"),
     [
         param(ibis.coalesce(5, None, 4), 5, id="first"),
-        param(ibis.coalesce(ibis.NA, 4, ibis.NA), 4, id="second"),
-        param(ibis.coalesce(ibis.NA, ibis.NA, 3.14), 3.14, id="third"),
+        param(ibis.coalesce(ibis.null(), 4, ibis.null()), 4, id="second"),
+        param(ibis.coalesce(ibis.null(), ibis.null(), 3.14), 3.14, id="third"),
     ],
 )
 def test_coalesce(con, expr, expected):
@@ -360,12 +360,12 @@ def test_coalesce(con, expr, expected):
 @pytest.mark.parametrize(
     ("expr", "expected"),
     [
-        param(ibis.coalesce(ibis.NA, ibis.NA), None, id="all_null"),
+        param(ibis.coalesce(ibis.null(), ibis.null()), None, id="all_null"),
         param(
             ibis.coalesce(
-                ibis.NA.cast("int8"),
-                ibis.NA.cast("int8"),
-                ibis.NA.cast("int8"),
+                ibis.null().cast("int8"),
+                ibis.null().cast("int8"),
+                ibis.null().cast("int8"),
             ),
             None,
             id="all_nulls_with_all_cast",
@@ -377,7 +377,7 @@ def test_coalesce_all_na(con, expr, expected):
 
 
 def test_coalesce_all_na_double(con):
-    expr = ibis.coalesce(ibis.NA, ibis.NA, ibis.NA.cast("double"))
+    expr = ibis.coalesce(ibis.null(), ibis.null(), ibis.null().cast("double"))
     assert np.isnan(con.execute(expr))
 
 
@@ -815,14 +815,14 @@ def test_first_last_value(alltypes, df, func, expected_index):
 def test_null_column(alltypes):
     t = alltypes
     nrows = t.count().execute()
-    expr = t.mutate(na_column=ibis.NA).na_column
+    expr = t.mutate(na_column=ibis.null()).na_column
     result = expr.execute()
     tm.assert_series_equal(result, pd.Series([None] * nrows, name="na_column"))
 
 
 def test_null_column_union(alltypes, df):
     t = alltypes
-    s = alltypes[["double_col"]].mutate(string_col=ibis.NA.cast("string"))
+    s = alltypes[["double_col"]].mutate(string_col=ibis.null().cast("string"))
     expr = t[["double_col", "string_col"]].union(s)
     result = expr.execute()
     nrows = t.count().execute()
diff --git a/ibis/backends/risingwave/tests/test_functions.py b/ibis/backends/risingwave/tests/test_functions.py
index c63f2bd64..89c012e7f 100644
--- a/ibis/backends/risingwave/tests/test_functions.py
+++ b/ibis/backends/risingwave/tests/test_functions.py
@@ -166,7 +166,7 @@ def test_regexp(con, expr, expected):
 @pytest.mark.parametrize(
     ("expr", "expected"),
     [
-        param(ibis.NA.fill_null(5), 5, id="filled"),
+        param(ibis.null().fill_null(5), 5, id="filled"),
         param(L(5).fill_null(10), 5, id="not_filled"),
         param(L(5).nullif(5), None, id="nullif_null"),
         param(L(10).nullif(5), 10, id="nullif_not_null"),
@@ -180,8 +180,8 @@ def test_fill_null_nullif(con, expr, expected):
     ("expr", "expected"),
     [
         param(ibis.coalesce(5, None, 4), 5, id="first"),
-        param(ibis.coalesce(ibis.NA, 4, ibis.NA), 4, id="second"),
-        param(ibis.coalesce(ibis.NA, ibis.NA, 3.14), 3.14, id="third"),
+        param(ibis.coalesce(ibis.null(), 4, ibis.null()), 4, id="second"),
+        param(ibis.coalesce(ibis.null(), ibis.null(), 3.14), 3.14, id="third"),
     ],
 )
 def test_coalesce(con, expr, expected):
@@ -191,12 +191,12 @@ def test_coalesce(con, expr, expected):
 @pytest.mark.parametrize(
     ("expr", "expected"),
     [
-        param(ibis.coalesce(ibis.NA, ibis.NA), None, id="all_null"),
+        param(ibis.coalesce(ibis.null(), ibis.null()), None, id="all_null"),
         param(
             ibis.coalesce(
-                ibis.NA.cast("int8"),
-                ibis.NA.cast("int8"),
-                ibis.NA.cast("int8"),
+                ibis.null().cast("int8"),
+                ibis.null().cast("int8"),
+                ibis.null().cast("int8"),
             ),
             None,
             id="all_nulls_with_all_cast",
@@ -208,7 +208,7 @@ def test_coalesce_all_na(con, expr, expected):
 
 
 def test_coalesce_all_na_double(con):
-    expr = ibis.coalesce(ibis.NA, ibis.NA, ibis.NA.cast("double"))
+    expr = ibis.coalesce(ibis.null(), ibis.null(), ibis.null().cast("double"))
     assert np.isnan(con.execute(expr))
 
 
@@ -595,7 +595,7 @@ def test_first_last_value(alltypes, df, func, expected_index):
 def test_null_column(alltypes):
     t = alltypes
     nrows = t.count().execute()
-    expr = t.mutate(na_column=ibis.NA).na_column
+    expr = t.mutate(na_column=ibis.null()).na_column
     result = expr.execute()
     tm.assert_series_equal(result, pd.Series([None] * nrows, name="na_column"))
 
diff --git a/ibis/backends/sqlite/tests/test_client.py b/ibis/backends/sqlite/tests/test_client.py
index cafea0abe..d7d5def38 100644
--- a/ibis/backends/sqlite/tests/test_client.py
+++ b/ibis/backends/sqlite/tests/test_client.py
@@ -47,7 +47,7 @@ def test_builtin_agg_udf(con):
     def total(x) -> float:
         """Totally total."""
 
-    expr = total(con.tables.functional_alltypes.limit(2).select(n=ibis.NA).n)
+    expr = total(con.tables.functional_alltypes.limit(2).select(n=ibis.null()).n)
     result = con.execute(expr)
     assert result == 0.0
 
diff --git a/ibis/backends/tests/sql/test_sql.py b/ibis/backends/tests/sql/test_sql.py
index 1f9e95542..6f1d11637 100644
--- a/ibis/backends/tests/sql/test_sql.py
+++ b/ibis/backends/tests/sql/test_sql.py
@@ -121,7 +121,7 @@ def test_coalesce(functional_alltypes, snapshot):
     d = functional_alltypes.double_col
     f = functional_alltypes.float_col
 
-    expr = ibis.coalesce((d > 30).ifelse(d, ibis.NA), ibis.NA, f).name("tmp")
+    expr = ibis.coalesce((d > 30).ifelse(d, ibis.null()), ibis.null(), f).name("tmp")
     snapshot.assert_match(to_sql(expr.name("tmp")), "out.sql")
 
 
diff --git a/ibis/backends/tests/test_generic.py b/ibis/backends/tests/test_generic.py
index 6db2eb4a6..cf7c1f1e3 100644
--- a/ibis/backends/tests/test_generic.py
+++ b/ibis/backends/tests/test_generic.py
@@ -118,7 +118,7 @@ def test_boolean_literal(con, backend):
 @pytest.mark.parametrize(
     ("expr", "expected"),
     [
-        param(ibis.NA.fill_null(5), 5, id="na_fill_null"),
+        param(ibis.null().fill_null(5), 5, id="na_fill_null"),
         param(ibis.literal(5).fill_null(10), 5, id="non_na_fill_null"),
         param(ibis.literal(5).nullif(5), None, id="nullif_null"),
         param(ibis.literal(10).nullif(5), 10, id="nullif_not_null"),
@@ -159,7 +159,10 @@ def test_scalar_fill_null_nullif(con, expr, expected):
             id="nan_col",
         ),
         param(
-            "none_col", ibis.NA.cast("float64"), methodcaller("isnull"), id="none_col"
+            "none_col",
+            ibis.null().cast("float64"),
+            methodcaller("isnull"),
+            id="none_col",
         ),
     ],
 )
@@ -224,8 +227,8 @@ def test_column_fill_null(backend, alltypes, value):
     ("expr", "expected"),
     [
         param(ibis.coalesce(5, None, 4), 5, id="generic"),
-        param(ibis.coalesce(ibis.NA, 4, ibis.NA), 4, id="null_start_end"),
-        param(ibis.coalesce(ibis.NA, ibis.NA, 3.14), 3.14, id="non_null_last"),
+        param(ibis.coalesce(ibis.null(), 4, ibis.null()), 4, id="null_start_end"),
+        param(ibis.coalesce(ibis.null(), ibis.null(), 3.14), 3.14, id="non_null_last"),
     ],
 )
 def test_coalesce(con, expr, expected):
@@ -539,9 +542,9 @@ def test_drop_null_table(backend, alltypes, how, subset):
     is_four = alltypes.int_col == 4
 
     table = alltypes.mutate(
-        col_1=is_two.ifelse(ibis.NA, alltypes.float_col),
-        col_2=is_four.ifelse(ibis.NA, alltypes.float_col),
-        col_3=(is_two | is_four).ifelse(ibis.NA, alltypes.float_col),
+        col_1=is_two.ifelse(ibis.null(), alltypes.float_col),
+        col_2=is_four.ifelse(ibis.null(), alltypes.float_col),
+        col_3=(is_two | is_four).ifelse(ibis.null(), alltypes.float_col),
     ).select("col_1", "col_2", "col_3")
 
     table_pandas = table.execute()
@@ -931,7 +934,7 @@ def test_logical_negation_column(backend, alltypes, df, op):
     [("int64", 0, 1), ("float64", 0.0, 1.0)],
 )
 def test_zero_ifnull_literals(con, dtype, zero, expected):
-    assert con.execute(ibis.NA.cast(dtype).fill_null(0)) == zero
+    assert con.execute(ibis.null().cast(dtype).fill_null(0)) == zero
     assert con.execute(ibis.literal(expected, type=dtype).fill_null(0)) == expected
 
 
diff --git a/ibis/backends/tests/test_map.py b/ibis/backends/tests/test_map.py
index 491efc46f..e637fb19a 100644
--- a/ibis/backends/tests/test_map.py
+++ b/ibis/backends/tests/test_map.py
@@ -606,7 +606,7 @@ def test_map_get_with_incompatible_value_different_kind(con):
 
 @mark_notimpl_risingwave_hstore
 @mark_notyet_postgres
-@pytest.mark.parametrize("null_value", [None, ibis.NA])
+@pytest.mark.parametrize("null_value", [None, ibis.null()])
 def test_map_get_with_null_on_not_nullable(con, null_value):
     map_type = dt.Map(dt.string, dt.Int16(nullable=False))
     value = ibis.literal({"A": 1000, "B": 2000}).cast(map_type)
@@ -615,7 +615,7 @@ def test_map_get_with_null_on_not_nullable(con, null_value):
     assert pd.isna(result)
 
 
-@pytest.mark.parametrize("null_value", [None, ibis.NA])
+@pytest.mark.parametrize("null_value", [None, ibis.null()])
 @pytest.mark.notyet(
     ["flink"], raises=Py4JJavaError, reason="Flink cannot handle typeless nulls"
 )
diff --git a/ibis/backends/tests/test_string.py b/ibis/backends/tests/test_string.py
index 7761253eb..423b116e4 100644
--- a/ibis/backends/tests/test_string.py
+++ b/ibis/backends/tests/test_string.py
@@ -923,7 +923,7 @@ def test_levenshtein(con, right):
     "expr",
     [
         param(ibis.case().when(True, "%").end(), id="case"),
-        param(ibis.ifelse(True, "%", ibis.NA), id="ifelse"),
+        param(ibis.ifelse(True, "%", ibis.null()), id="ifelse"),
     ],
 )
 def test_no_conditional_percent_escape(con, expr):
diff --git a/ibis/backends/tests/test_struct.py b/ibis/backends/tests/test_struct.py
index c791318f1..6a7429a6c 100644
--- a/ibis/backends/tests/test_struct.py
+++ b/ibis/backends/tests/test_struct.py
@@ -73,7 +73,7 @@ _STRUCT_LITERAL = ibis.struct(
     _SIMPLE_DICT,
     type="struct<a: int64, b: string, c: float64>",
 )
-_NULL_STRUCT_LITERAL = ibis.NA.cast("struct<a: int64, b: string, c: float64>")
+_NULL_STRUCT_LITERAL = ibis.null().cast("struct<a: int64, b: string, c: float64>")
 
 
 @pytest.mark.notimpl(["postgres", "risingwave"])
diff --git a/ibis/backends/tests/test_window.py b/ibis/backends/tests/test_window.py
index 88ad8e55a..b0684a278 100644
--- a/ibis/backends/tests/test_window.py
+++ b/ibis/backends/tests/test_window.py
@@ -637,7 +637,7 @@ def test_simple_ungrouped_unbound_following_window(
 @pytest.mark.xfail_version(datafusion=["datafusion==35"])
 def test_simple_ungrouped_window_with_scalar_order_by(alltypes):
     t = alltypes[alltypes.double_col < 50].order_by("id")
-    w = ibis.window(rows=(0, None), order_by=ibis.NA)
+    w = ibis.window(rows=(0, None), order_by=ibis.null())
     expr = t.double_col.sum().over(w).name("double_col")
     # hard to reproduce this in pandas, so just test that it actually executes
     expr.execute()
diff --git a/ibis/tests/expr/test_table.py b/ibis/tests/expr/test_table.py
index be1a20cfa..4dab6aee3 100644
--- a/ibis/tests/expr/test_table.py
+++ b/ibis/tests/expr/test_table.py
@@ -569,7 +569,7 @@ rand = ibis.random()
 @pytest.mark.parametrize(
     ("key", "expected"),
     [
-        param(ibis.NA, ibis.NA.op(), id="na"),
+        param(ibis.null(), ibis.null().op(), id="na"),
         param(rand, rand.op(), id="random"),
         param(1.0, ibis.literal(1.0).op(), id="float"),
         param(ibis.literal("a"), ibis.literal("a").op(), id="string"),
diff --git a/ibis/tests/expr/test_timestamp.py b/ibis/tests/expr/test_timestamp.py
index 454355458..45b8972d5 100644
--- a/ibis/tests/expr/test_timestamp.py
+++ b/ibis/tests/expr/test_timestamp.py
@@ -106,7 +106,7 @@ def test_greater_comparison_pandas_timestamp(alltypes):
 
 def test_timestamp_precedence():
     ts = ibis.literal(datetime.now())
-    highest_type = rlz.highest_precedence_dtype([ibis.NA.op(), ts.op()])
+    highest_type = rlz.highest_precedence_dtype([ibis.null().op(), ts.op()])
     assert highest_type == dt.timestamp
 
 
diff --git a/ibis/tests/expr/test_value_exprs.py b/ibis/tests/expr/test_value_exprs.py
index cdd60788d..e7b573760 100644
--- a/ibis/tests/expr/test_value_exprs.py
+++ b/ibis/tests/expr/test_value_exprs.py
@@ -352,7 +352,7 @@ def test_notnull(table):
 
 @pytest.mark.parametrize(
     "value",
-    [None, ibis.NA, ibis.literal(None, type="int32")],
+    [None, ibis.null(), ibis.literal(None, type="int32")],
     ids=["none", "NA", "typed-null"],
 )
 def test_null_eq_and_ne(table, value):
@@ -648,7 +648,7 @@ def test_or_(table):
 
 def test_null_column():
     t = ibis.table([("a", "string")], name="t")
-    s = t.mutate(b=ibis.NA)
+    s = t.mutate(b=ibis.null())
     assert s.b.type() == dt.null
     assert isinstance(s.b, ir.NullColumn)
 
@@ -657,8 +657,8 @@ def test_null_column_union():
     s = ibis.table([("a", "string"), ("b", "double")])
     t = ibis.table([("a", "string")])
     with pytest.raises(ibis.common.exceptions.RelationError):
-        s.union(t.mutate(b=ibis.NA))  # needs a type
-    assert s.union(t.mutate(b=ibis.NA.cast("double"))).schema() == s.schema()
+        s.union(t.mutate(b=ibis.null()))  # needs a type
+    assert s.union(t.mutate(b=ibis.null().cast("double"))).schema() == s.schema()
 
 
 def test_string_compare_numeric_array(table):
@@ -843,12 +843,12 @@ def test_substitute_dict():
     )
     assert_equal(result, expected)
 
-    result = table.foo.substitute(subs, else_=ibis.NA)
+    result = table.foo.substitute(subs, else_=ibis.null())
     expected = (
         ibis.case()
         .when(table.foo == "a", "one")
         .when(table.foo == "b", table.bar)
-        .else_(ibis.NA)
+        .else_(ibis.null())
         .end()
     )
     assert_equal(result, expected)
@@ -926,7 +926,7 @@ def test_generic_value_api_no_arithmetic(value, operation):
     ("value", "expected"), [(5, dt.int8), (5.4, dt.double), ("abc", dt.string)]
 )
 def test_fill_null_null(value, expected):
-    assert ibis.NA.fill_null(value).type().equals(expected)
+    assert ibis.null().fill_null(value).type().equals(expected)
 
 
 @pytest.mark.parametrize(
@@ -1229,7 +1229,7 @@ def test_map_get_with_incompatible_value_different_kind():
     assert value.get("C", 3.0).type() == dt.float64
 
 
-@pytest.mark.parametrize("null_value", [None, ibis.NA])
+@pytest.mark.parametrize("null_value", [None, ibis.null()])
 def test_map_get_with_null_on_not_nullable(null_value):
     map_type = dt.Map(dt.string, dt.Int16(nullable=False))
     value = ibis.literal({"A": 1000, "B": 2000}).cast(map_type)
@@ -1238,14 +1238,14 @@ def test_map_get_with_null_on_not_nullable(null_value):
     assert expr.type() == dt.Int16(nullable=True)
 
 
-@pytest.mark.parametrize("null_value", [None, ibis.NA])
+@pytest.mark.parametrize("null_value", [None, ibis.null()])
 def test_map_get_with_null_on_nullable(null_value):
     value = ibis.literal({"A": 1000, "B": None})
     result = value.get("C", null_value)
     assert result.type().nullable
 
 
-@pytest.mark.parametrize("null_value", [None, ibis.NA])
+@pytest.mark.parametrize("null_value", [None, ibis.null()])
 def test_map_get_with_null_on_null_type_with_null(null_value):
     value = ibis.literal({"A": None, "B": None})
     result = value.get("C", null_value)
@@ -1378,13 +1378,13 @@ def test_repr_list_of_lists_in_table():
 @pytest.mark.parametrize(
     ("expr", "expected_type"),
     [
-        (ibis.coalesce(ibis.NA, 1), dt.int8),
-        (ibis.coalesce(1, ibis.NA), dt.int8),
-        (ibis.coalesce(ibis.NA, 1000), dt.int16),
-        (ibis.coalesce(ibis.NA), dt.null),
-        (ibis.coalesce(ibis.NA, ibis.NA), dt.null),
+        (ibis.coalesce(ibis.null(), 1), dt.int8),
+        (ibis.coalesce(1, ibis.null()), dt.int8),
+        (ibis.coalesce(ibis.null(), 1000), dt.int16),
+        (ibis.coalesce(ibis.null()), dt.null),
+        (ibis.coalesce(ibis.null(), ibis.null()), dt.null),
         (
-            ibis.coalesce(ibis.NA, ibis.NA.cast("array<string>")),
+            ibis.coalesce(ibis.null(), ibis.null().cast("array<string>")),
             dt.Array(dt.string),
         ),
     ],
@@ -1508,14 +1508,14 @@ def test_deferred_r_ops(op_name, expected_left, expected_right):
 @pytest.mark.parametrize(
     ("expr_fn", "expected_type"),
     [
-        (lambda t: ibis.ifelse(t.a == 1, t.b, ibis.NA), dt.string),
+        (lambda t: ibis.ifelse(t.a == 1, t.b, ibis.null()), dt.string),
         (lambda t: ibis.ifelse(t.a == 1, t.b, t.a.cast("string")), dt.string),
         (
             lambda t: ibis.ifelse(t.a == 1, t.b, t.a.cast("!string")),
             dt.string.copy(nullable=False),
         ),
-        (lambda _: ibis.ifelse(True, ibis.NA, ibis.NA), dt.null),
-        (lambda _: ibis.ifelse(False, ibis.NA, ibis.NA), dt.null),
+        (lambda _: ibis.ifelse(True, ibis.null(), ibis.null()), dt.null),
+        (lambda _: ibis.ifelse(False, ibis.null(), ibis.null()), dt.null),
     ],
 )
 def test_non_null_with_null_precedence(expr_fn, expected_type):
diff --git a/ibis/tests/expr/test_window_frames.py b/ibis/tests/expr/test_window_frames.py
index 2e88f2c2c..5560a3608 100644
--- a/ibis/tests/expr/test_window_frames.py
+++ b/ibis/tests/expr/test_window_frames.py
@@ -234,7 +234,7 @@ def test_window_api_supports_value_expressions(t):
 
 
 def test_window_api_supports_scalar_order_by(t):
-    window = ibis.window(order_by=ibis.NA)
+    window = ibis.window(order_by=ibis.null())
     expr = t.a.sum().over(window).op()
     expected = ops.WindowFunction(
         t.a.sum(),
@@ -242,7 +242,7 @@ def test_window_api_supports_scalar_order_by(t):
         start=None,
         end=None,
         group_by=(),
-        order_by=(ibis.NA.op(),),
+        order_by=(ibis.null().op(),),
     )
     assert expr == expected
 
diff --git a/ibis/tests/test_api.py b/ibis/tests/test_api.py
index fc672a4af..b21f566f6 100644
--- a/ibis/tests/test_api.py
+++ b/ibis/tests/test_api.py
@@ -26,7 +26,7 @@ import ibis
 
 # def test_public_backend_methods():
 #     public = {m for m in dir(ibis.sqlite) if not m.startswith("_")}
-#     assert public == {"connect", "compile", "has_operation", "add_operation", "name"}
+#     assert public == {"connect", "compile", "has_operation", "name"}
 
 
 def test_missing_backend():
@@ -69,3 +69,10 @@ import sys
 assert "{module}" not in sys.modules
 """
     subprocess.run([sys.executable, "-c", script], check=True)
+
+
+def test_ibis_na_deprecation_warning():
+    with pytest.warns(
+        DeprecationWarning, match="The 'ibis.NA' constant is deprecated as of v9.1"
+    ):
+        assert ibis.NA is ibis.null()