Title: Processes with python multiprocessing not ending
Tags: python,queue,python-multiprocessing

Question: Trying to get multiprocessing working correctly. Have review lots of posts in Stack overflow but none seem to fit my issue. I have a batch of pdfs that I am extracting the text data from. Using multiprocessing with queue to speed up the process. My script starts the processes and extracts the text from the pdfs.
There is a print statement in the main.py that is never executed saying that there are processes that never finish.
print(f'Finished in {round(finish - start, 2)} seconds(s)')

My main.py read as
import multiprocessing
from multiprocessing import freeze_support, Queue  # Add this line to support Windows platform
import ExtractPDFData as eD
import time
import os


if __name__ == '__main__':
    freeze_support()
    files_to_process = []
    pdf_directory = 'PDFData'

    for each_pdf_file in os.listdir(pdf_directory):
        if each_pdf_file.endswith('.pdf'):
            files_to_process.append(os.path.join(pdf_directory, each_pdf_file))

    start = time.perf_counter()
    q = Queue()
    current_processes_array = []

    for pdf_file in files_to_process:
        p = multiprocessing.Process(target=eD.pdf_process, args=(pdf_file, q))
        p.start()
        current_processes_array.append(p)

    print(f"Number of active children: {len(multiprocessing.active_children())}")

    for process in current_processes_array:
        process.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish - start, 2)} seconds(s)')

    with open('OutputData/halo_data.txt', 'w', encoding='UTF-8', errors='ignore') as add_text_to_file:
        while not q.empty():
            add_text_to_file.write(q.get())
        add_text_to_file.close()

    with open('OutputData/halo_data.txt', 'r', encoding='UTF-8', errors='ignore') as f:
        print(f'Number of char in the input file: {len(f.read())}')

The attaching ExtractedPDFData.py is
from pypdf import PdfReader
import os


# Extracts text from PDF file
def pdf_process(pdf_path, q):
    with open(pdf_path, 'rb') as f:
        reader = PdfReader(f)
        cleaned_pages_array = []

        for page in range(len(reader.pages)):
            # Extract the text from each page
            page_of_text = reader.pages[page].extract_text()
            # Convert to lowercase
            page_of_text = page_of_text.lower()
            # Remove non-alphanumeric characters and extra whitespaces
            cleaned_pages_array.append(page_of_text)

        print('Number of pages in the array: ', len(cleaned_pages_array))
        text = " ".join(cleaned_pages_array)
        q.put(text)

        print("Have written to file " + pdf_path)

        print(f"Worker process ID: {os.getpid()}, which comes from the parent {os.getppid()}")

Answer: I took the point that the 'queue' needs to be consumed before 'join()'. Tested that premise but my script still didn't work. The premise however was correct but is seems it is what I putting in the 'queue' that is causing the issue - 'queue' and 'process' with big text files isn't that right mix.
When I altered my code to 'put' string values into the 'queue' instead of big 'text' files and the script ran successfully. I then placed the big text files into an object and 'put' those object reference values to the 'queue', but that didn't work. I really thought that would be successful.
So small values work but not large text files, or reference values.
Next step is to look at 'Pooling'.