convert_and_split.py

This script was designed to transform the data I was given into workable chunks. Namely, breaking up ~200-page PDF documents containing 100 news articles each, and similarly sized .docx files of press releases, into individual text files named by the unique document ID of each entry. However, the large files were not all formatted the same way, so I had to use various different parts of the individual entries as delimeters, as marked by the command line flags. This program uses concurrency because converting files from PDF to text using the pdfminer module takes several minutes each.
This Script's Operations:
1. Use regular expressions to make a list of all the file names in the given directory that match the pattern for press releases / articles
2. For each file (either iteratively or concurrently):
a) Convert file from .pdf/.docx to .txt and save .txt file (for reference and troubleshooting)
b) Split .txt file into individual articles/press releases, and save under the document ID