Package: linkrot
IntroductionScans PDFs for links written in plaintext and checks if they are active or returns an error code. It then generates a report of its findings. Extract references (PDF, URL, DOI, arXiv) and metadata from a PDF.Check out our sister project,Rotting Research, for a web app implementation of this project.FeaturesExtract references and metadata from a given PDF.Detects PDF, URL, arXiv and DOI references.Archives valid links using Internet Archive's Wayback Machine (using the -a flag).Checks for valid SSL certificate.Find broken hyperlinks (using the -c flag).Output as text or JSON (using the -j flag).Extract the PDF text (using the --text flag).Use as command-line tool or Python package.Works with local and online PDFs.InstallationGrab a copy of the code with pip:pipinstalllinkrotUsagelinkrot can be used to extract info from a PDF in two ways:Command line/Terminal toollinkrotPython libraryimport linkrot1. Command Line/Terminal toollinkrot[pdf-file-or-url]Run linkrot -h to see the help output:linkrot-husage:linkrot[-h][-dOUTPUT_DIRECTORY][-c][-j][-v][-t][-oOUTPUT_FILE][--version]pdfExtract metadata and references from a PDF, and optionally download all
referenced PDFs.Argumentspositional arguments:pdf                   (Filename or URL of a PDF file)optional arguments:-h, --help            (Show this help message and exit)  
-d OUTPUT_DIRECTORY,  --download-pdfs OUTPUT_DIRECTORY (Download all referenced PDFs into specified directory)  
-c, --check-links     (Check for broken links)  
-j, --json            (Output infos as JSON (instead of plain text))  
-v, --verbose         (Print all references (instead of only PDFs))  
-t, --text            (Only extract text (no metadata or references))  
-a, --archive	  (Archive actvice links)
-o OUTPUT_FILE,        --output-file OUTPUT_FILE (Output to specified file instead of console)  
--version             (Show program's version number and exit)PDF SamplesFor testing purposes, you can find PDF samples inshared MEGAfolder](https://mega.nz/folder/uwBxVSzS#lpBtSz49E9dqHtmrQwp0Ig).ExamplesExtract text to console.linkrothttps://example.com/example.pdf-tExtract text to filelinkrothttps://example.com/example.pdf-t-opdf-text.txtCheck Linkslinkrothttps://example.com/example.pdf-c2. Main Python LibraryImport the library:importlinkrotCreate an instance of the linkrot class like so:pdf=linkrot.linkrot("filename-or-url.pdf")#pdf is the instance of the linkrot classNow the following function can be used to extract specific data from the pdf:get_metadata()Arguments: NoneUsage:metadata=pdf.get_metadata()#pdf is the instance of the linkrot classReturn type: Dictionary<class 'dict'>Information Provided: All metadata, secret metadata associated with the PDF including Creation date, Creator, Title, etc...get_text()Arguments: NoneUsage:text=pdf.get_text()#pdf is the instance of the linkrot classReturn type: String<class 'str'>Information Provided: The entire content of the PDF in string form.get_references(reftype=None, sort=False)Arguments:reftype: The type of reference that is needed 
	 values: 'pdf', 'url', 'doi', 'arxiv'. 
	 default: Provides all reference types.

sort: Whether reference should be sorted or not
      values: True or False. 
      default: Is not sorted.Usage:references_list=pdf.get_references()#pdf is the instance of the linkrot classReturn type: Set<class 'set'>of<linkrot.backends.Reference object>linkrot.backends.Reference object has 3 member variables:
- ref: actual URL/PDF/DOI/ARXIV
- reftype: type of reference
- page: page on which it was referencedInformation Provided: All references with their corresponding type and page number.get_references_as_dict(reftype=None, sort=False)Arguments:reftype: The type of reference that is needed 
	 values: 'pdf', 'url', 'doi', 'arxiv'. 
	 default: Provides all reference types.

sort: Whether reference should be sorted or not
      values: True or False. 
      default: Is not sorted.Usage:references_dict=pdf.get_references_as_dict()#pdf is the instance of the linkrot classReturn type: Dictionary<class 'dict'>with keys 'pdf', 'url', 'doi', 'arxiv' that each have a list<class 'list'>of refs of that type.Information Provided: All references in their corresponding type list.download_pdfs(target_dir)Arguments:target_dir: The path of the directory to which the reference PDFs should be downloadedUsage:pdf.download_pdfs("target-directory")#pdf is the instance of the linkrot classReturn type: NoneInformation Provided: Downloads all the reference PDFs to the specified directory.3. Linkrot downloader functionsImport:fromlinkrot.downloaderimportsanitize_url,get_status_code,check_refssanitize_url(url)Arguments:url: The url to be sanitized.Usage:new_url=sanitize_url(old_url)Return type: String<class 'str'>Information Provided: URL is prefixed with 'http://' if it was not before and makes sure it is in utf-8 format.get_status_code(url)Arguments:url: The url to be checked for its status.Usage:status_code=get_status_code(url)Return type: String<class 'str'>Information Provided: Checks if the URL is active or broken.check_refs(refs, verbose=True, max_threads=MAX_THREADS_DEFAULT)Arguments:refs: set of linkrot.backends.Reference objects
verbose: whether it should print every reference with its code or just the summary of the link checker
max_threads: number of threads for multithreadingUsage:check_refs(pdf.get_references())#pdf is the instance of the linkrot classReturn type: NoneInformation Provided: Prints references with their status code and a summary of all the broken/active links on terminal.4. Linkrot extractor functionsImport:fromlinkrot.extractorimportextract_urls,extract_doi,extract_arxivGet pdf text:text=pdf.get_text()#pdf is the instance of the linkrot classextract_urls(text)Arguments:text: String of text to extract urls fromUsage:urls=extract_urls(text)Return type: Set<class 'set'>of URLsInformation Provided: All URLs in the textextract_arxiv(text)Arguments:text: String of text to extract arXivs fromUsage:arxiv=extract_arxiv(text)Return type: Set<class 'set'>of arxivsInformation Provided: All arXivs in the textextract_doi(text)Arguments:text: String of text to extract DOIs fromUsage:doi=extract_doi(text)Return type: Set<class 'set'>of DOIsInformation Provided: All DOIs in the textCode of ConductTo view our code of conduct please visit ourCode of Conduct page.LicenseThis program is licensed with anGPLv3 License.