Package: url-metadata
This is currently not perfect and in development, so expect changes to the API/interface. It aims to walk the line between extracting enough text/data for it to be useful, but no so much that it takes enormous amounts of space.Current TODOs:Improve CLI interface to match all functionsImprove HTML/text parsing (see#6)Add more sites using theabstract interface, to get more info from sites I use commonlyAdd a preprocessing step to the sites abstract interface/URLMetadataCache functions, which 'corrects' URLs, to avoid hash mismatchesA cache which saves URL metadata and summarizes contentThis is meant to provide more context to any of my tools which use URLs. If Iwatched some youtube videoand I have a URL, I'd like to have the subtitles for it, so I can do a text-search over all the videos I've watched. If Iread an article, I want the article text! This requests, parses and abstracts away that data for me locally, so I can just do:>>>fromurl_metadataimportmetadata>>>m=metadata("https://pypi.org/project/beautifulsoup4/")>>>len(m.info["images"])46>>>m.info["title"]'beautifulsoup4'>>>m.text_summary[:57]"Beautiful Soup is a library that makes it easy to scrape"If I ever request the same URL again, that info is grabbed from a local directory cache instead.InstallationRequirespython3.7+To install with pip, run:pip install url_metadataThis uses:lassieto get generic metadata; the title, description, opengraph information, links to images/videos on the pagereadabilityto convert HTML to a summary of the HTML content.bs4to convert the parsed HTML to text (to allow for nicer text searching)youtube_subtitles_downloaderto get manual/autogenerated captions (converted to a.srtfile) from Youtube URLs.Usage:The CLI interface provides some utility commands to get/list information from the cache.$ url_metadata --help
Usage: url_metadata [OPTIONS] COMMAND [ARGS]...

Options:
  --cache-dir PATH          Override default cache directory location
  --debug / --no-debug      Increase log verbosity
  --sleep-time INTEGER      How long to sleep between requests
  --skip-subtitles          Don't attempt to download subtitles
  --subtitle-language TEXT  Subtitle language for Youtube captions
  --help                    Show this message and exit.

Commands:
  cachedir  Prints the location of the local cache directory
  export    Print all cached information as JSON
  get       Get information for one or more URLs Prints results as JSON
  list      List all cached URLsIn Python, this can be configured by using theurl_metadata.URLMetadataCacheclass:url_metadata.URLMetadataCache(loglevel:int=30,subtitle_language:str='en',sleep_time:int=5,skip_subtitles:bool=False,cache_dir:Optional[str,pathlib.Path]=None):"""Main interface to the librarysubtitle_language: for youtube subtitle requestssleep_time: time to wait between HTTP requestsskip_subtitles: don't attempt to download youtube subtitlescache_dir: location the store cached datauses default user cache directory if not provided"""get(self,url:str)->url_metadata.model.Metadata"""Gets metadata/summary for a URLSave the parsed information in a local data directoryIf the URL already has cached data locally, returns that instead"""get_cache_dir(self,url:str)->Optional[str]"""If this URL is in cache, returns the location of the cache directoryReturns None if it couldn't find a matching directory"""in_cache(self,url:str)->bool"""Returns True if the URL already has cached information"""request_data(self,url:str)->url_metadata.model.Metadata"""Given a URL:If this is a youtube URL, this requests youtube subtitlesUses lassie to grab metadataParses the HTML text with readablityuses bs4 to parse that text into a plaintext summary"""For example:importloggingfromurl_metadataimportURLMetadataCache# make requests every 2 seconds# debug logs# save to a folder in my home directorycache=URLMetadataCache(loglevel=logging.DEBUG,sleep_time=2,cache_dir="~/mydata")c=cache.get("https://github.com/seanbreckenridge/url_metadata")# just request information, don't read/save to cachedata=cache.request_data("https://www.wikipedia.org/")CLI ExamplesThegetcommand emitsJSON, so it could with other tools (e.g.jq) used like:$url_metadataget"https://click.palletsprojects.com/en/7.x/arguments/"\|jq-r'.[] | .text_summary'|head-n5
Arguments
Argumentsworksimilarlytooptionsbutarepositional.
Theyalsoonlysupportasubsetofthefeaturesofoptionsduetotheir
syntacticalnature.Clickwillalsonotattempttodocumentargumentsforyouandwantsyoutodocumentthemmanually$url_metadataexport|jq-r'.[] | .info | .title'seanbreckenridge/youtube_subtitles_downloader
Arguments—ClickDocumentation(7.x)$url_metadatalist--location
/home/sean/.local/share/url_metadata/data/b/a/a/c8e05501857a3c7d2d1a94071c68e/000
/home/sean/.local/share/url_metadata/data/9/4/4/1c380792a3d62302e1137850d177b/000# to make a backup of the cache directory$tar-cvzfurl_metadata.tar.gz"$(url_metadatacachedir)"Accessible through theurl_metadatascript andpython3 -m url_metadataImplementation NotesThis stores all of this information as individual files in a cache directory (usingappdirs). In particular, itMD5hashes the URL and stores information like:.
└── 7
    └── b
        └── 0
            └── d952fd7265e8e4cf10b351b6f8932
                └── 000
                    ├── epoch_timestamp.txt
                    ├── key
                    ├── metadata.json
                    ├── subtitles.srt
                    ├── summary.html
                    └── summary.txtYou're free to delete any of the directories in the cache if you want, this doesn't maintain a strict index, it uses a hash of the URL and then searches for a matchingkeyfile. See commentsherefor implementation details.By default this waits 5 seconds between requests. Since all the info is cached, I use this by requesting all the info from one data source (e.g. my bookmarks, or videos I've watched recently) in a loop in the background, which saves all the information to my computer. The next time I do that same loop, it doesn't have to make any requests and it just grabs all the info from local cache.Originally created forHPI.Testinggit clone 'https://github.com/seanbreckenridge/url_metadata'
cd ./url_metadata
git submodule update --init
pip install '.[testing]'
mypy ./src/url_metadata/
pytest