Package: url-cache
This is currently very alpha and in development, so expect changes to the API/interface. It aims to walk the line between extracting enough text/data for it to be useful, but no so much that it takes enormous amounts of space.As it stands I'm sort of pessimistic this would ever be a silver bullet, getting useful info out of arbitrary HTML is hard, so you're sort of stuck writing parsers for each website you're interested in. However, I still use this frequently, especially as a cache for API information like describedbelowCurrent TODOs:Add more sites using theabstract interface, to get more info from sites I use commonly. Ideally, should be able to re-use common scraper/parsers/API interface libraries in python, instead of recreating everything from scratchCreate a (separate repo/project) daemon which handles configuring this and slowly requests things in the background as they become available through given sources; allow user to provide generators/inputs define include/exclude lists/regexes. Probably just integrate withpromnesiaso avoid duplicating the work of searching for URLs on diskInstallationRequirespython3.7+To install with pip, run:python3 -m pip install url_cacheAs this is still in development, for the latest changes install from git:python3 -m pip install git+https://github.com/seanbreckenridge/url_cacheRationaleA file system cache which saves URL metadata and summarizes contentThis is meant to provide more context to any of my tools which use URLs. If Iwatched some youtube videoand I have a URL, I'd like to have the subtitles for it, so I can do a text-search over all the videos I've watched. If Iread an article, I want the article text! This requests, parses and abstracts away that data for me locally, so I can do something like:>>>fromurl_cache.coreimportURLCache>>>u=URLCache()>>>data=u.get("https://sean.fish/")>>>data.metadata["images"][-1]{'src':'https://raw.githubusercontent.com/seanbreckenridge/glue/master/assets/screenshot.png','alt':'screenshot','type':'body_image','width':600}>>>data.metadata["description"]"sean.fish; Sean Breckenridge's Home Page"If I ever request that URL again, the information is grabbed from a local cache instead.Generally, this uses:lassieto get generic metadata; the title, description, opengraph information, links to images/videos on the page.readabilityto convert/compress HTML to a summary of the HTML content.Site-Specific Extractors:Youtube: to get manual/auto-generated captions (converted to a.srtfile) from Youtube URLsStackoverflow (Just a basic URL preprocessor to reduce the possibility of conflicts/duplicate data)MyAnimeList (usingJikan v4)This is meant to be extendible -- so its possible for you to write your own extractors/file loaders/dumpers (for new formats (e.g.srt)) for sites you use commonly and pass those tourl_cache.core.URLCacheto extract richer data for those sites. Otherwise, it saves the information fromlassieand the summarized HTML usingreadabilityfor each URL.To avoid scope creep, this probably won't support:Converting the HTML summary to text (use something like thelynxcommand below)Minimizing HTML - run something likefind ~/.local/share/url_cache/ -name '*.html' -exec <some tool/script that minimizes in place> \;instead -- the data is just stored in individual files in the data directoryUsage:In Python, this can be configured by using theurl_cache.core.URLCacheclass: For example:importloggingfromurl_cache.coreimportURLCache# make requests every 2 seconds# debug logs# save to a folder in my home directorycache=URLCache(loglevel=logging.DEBUG,sleep_time=2,cache_dir="~/Documents/urldata")c=cache.get("https://github.com/seanbreckenridge")# just request information, don't read/save to cachedata=cache.request_data("https://www.wikipedia.org/")For more information, seethe docsThe CLI interface provides some utility commands to get/list information from the cache.Usage: url_cache [OPTIONS] COMMAND [ARGS]...

Options:
  --cache-dir PATH                Override default cache directory location
  --debug / --no-debug            Increase log verbosity
  --sleep-time INTEGER            How long to sleep between requests
  --summarize-html / --no-summarize-html
                                  Use readability to summarize html. Otherwise
                                  saves the entire HTML document

  --skip-subtitles / --no-skip-subtitles
                                  Skip downloading Youtube Subtitles
  --subtitle-language TEXT        Subtitle language for Youtube Subtitles
  --help                          Show this message and exit.

Commands:
  cachedir  Prints the location of the local cache directory
  export    Print all cached information as JSON
  get       Get information for one or more URLs Prints results as JSON
  in-cache  Prints if a URL is already cached
  list      List all cached URLsAn environment variableURL_CACHE_DIRcan be set, which changes the default cache directory.API Cache ExamplesI've also successfully used this to cache responses from API results in some of my projects, by subclassing and overriding therequest_datafunction. I just make a request and return a summary, and it transparently caches the rest. See:albums/discogs_cachemy_feed/tmdbdbsentinel/metadataCLI ExamplesThegetcommand emitsJSON, so it could with other tools (e.g.jq) used like:$url_cacheget"https://click.palletsprojects.com/en/7.x/arguments/"|\jq-r'.[] | .html_summary'|lynx-stdin-dump|head-n5ArgumentsArgumentsworksimilarlyto[1]optionsbutarepositional.Theyalsoonlysupportasubsetofthefeaturesofoptionsduetotheirsyntacticalnature.Clickwillalsonotattempttodocumentarguments$url_cacheexport|jq-r'.[] | .metadata | .title'seanbreckenridge-Overview
Arguments—ClickDocumentation(7.x)url_cachelist--location
/home/sean/.local/share/url_cache/data/2/c/7/6284b2f664f381372fab3276449b2/000
/home/sean/.local/share/url_cache/data/7/5/1/70fc230cd88f32e475ff4087f81d9/000# to make a backup of the cache directory$tar-cvzfurl_cache.tar.gz"$(url_cachecachedir)"Accessible through theurl_cachescript andpython3 -m url_cacheImplementation NotesThis stores all of this information as individual files in a cache directory. In particular, itMD5hashes the URL and stores information like:.
└── a
    └── a
        └── e
            └── cf0118bb22340e18fff20f2db8abd
                └── 000
                    ├── data
                    │   └── subtitles.srt
                    ├── key
                    ├── metadata.json
                    └── timestamp.datetime.txtIn other words, this is a file system hash table which implements separate chaining.You're free to delete any of the directories in the cache if you want, this doesn't maintain a strict index, it uses a hash of the URL and then searches for a matchingkeyfile.By default this waits 5 seconds between requests. Since all the info is cached, I use this by requesting all the info from one data source (e.g. my bookmarks, or videos I've watched recently) in a loop in the background, which saves all the information to my computer. The next time I do that same loop, it doesn't have to make any requests and it just grabs all the info from local cache.Originally created forHPI.Testinggit clone 'https://github.com/seanbreckenridge/url_cache'
cd ./url_cache
pip install '.[testing]'
mypy ./src/url_cache
flake8 ./src/url_cache
pytest