Repository: GSA-TTS/searchgov-spider
Pull request #132: SRCH-5871: Added support for parsing PDFs (v2)
Author: login: igoristic
Created: 2025-03-14T23:12:00Z | Merged: 2025-03-18T15:30:02Z

**Local testing:**

STEP: 1
---
```
# WARNING: Some of these cmds will delete *all* persistent docker data. Use with caution!

# stop/remove all docker images:
docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

# remove all docker mounted/binded data:
docker system prune -a
docker volume prune -a

# start docker compose 
docker-compose up

# ...wait about a minute for docker to finish all the automation

# confirm plugins are installed:
curl --location 'http://localhost:9200/_cat/plugins'

# confirm default/dev elasticsearch index is created "development-i14y-documents-searchgov":
curl --location 'http://localhost:9200/_cat/indices'

# Now you can run spiders locally with "output_target": "elasticsearch"
# Also works if search-gov docker containers are already running
```

STEP: 2
---
Ingest a site with known pdf files, eg:
```
scrapy crawl domain_spider -a allowed_domains=cms.gov -a start_urls=https://www.cms.gov -a output_target=elasticsearch
```

STEP: 3
---
Go to Kibana > Dev Tools and confirm pdf documents are in elasticsearch via:
```
GET development-i14y-documents-searchgov/_search
{
  "sort": [
    {
      "_score": {
        "order": "desc"
      },
      "created_at": {
        "order": "desc"
      }
    }
  ],
  "size": 500,
  "_source": true,
  "query": {
    "bool": {
      "must": [
        {
          "bool": {
            "should": [
              {
                "query_string": {
                  "fields": [
                    "extension"
                  ],
                  "query": "pdf"
                }
              }
            ],
            "minimum_should_match": 1
          }
        }
      ]
    }
  }
}
```

#### Functionality Checks

- [x] You have merged the latest changes from the target branch (usually `main`) into your branch.

- [x] Your primary commit message is of the format **SRCH-#### \<description\>** matching the associated Jira ticket.

- [x] PR title is either of the format **SRCH-#### \<description\>** matching the associated Jira ticket (i.e. "SRCH-123 implement feature X"), or **Release - SRCH-####, SRCH-####, SRCH-####** matching the Jira ticket numbers in the release.

- [x] Automated checks pass. If Code Climate checks do not pass, explain reason for failures:

#### Process Checks

- [x] You have specified at least one "Reviewer".


Diff:
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
index 02565c3e..f6c9f5ae 100755
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -18,6 +18,8 @@ updates:
           - "newspaper4k*"
           - "cchardet"
           - "python-dotenv"
+          - "pypdf"
+          - "langdetect"
       testing-and-linting:
         patterns:
           - "freezegun"
diff --git a/search_gov_crawler/benchmark.py b/search_gov_crawler/benchmark.py
index 38685815..1b53a1a4 100755
--- a/search_gov_crawler/benchmark.py
+++ b/search_gov_crawler/benchmark.py
@@ -210,6 +210,14 @@ def benchmark_from_args(
         required=no_input_arg,
         choices=list(ALLOWED_CONTENT_TYPE_OUTPUT_MAP.keys()),
     )
+    parser.add_argument(
+        "-o",
+        "--output_target",
+        type=str,
+        help="Point the output of the crawls to a backend",
+        required=no_input_arg,
+        choices=list(ALLOWED_CONTENT_TYPE_OUTPUT_MAP.keys()),
+    )
     parser.add_argument(
         "-js",
         "--handle_js",
diff --git a/search_gov_crawler/elasticsearch/convert_html_i14y.py b/search_gov_crawler/elasticsearch/convert_html_i14y.py
index 00cfa6ee..e388041b 100644
--- a/search_gov_crawler/elasticsearch/convert_html_i14y.py
+++ b/search_gov_crawler/elasticsearch/convert_html_i14y.py
@@ -1,32 +1,15 @@
-import hashlib
-import os
-from datetime import UTC, datetime
-from urllib.parse import urlparse
-
 import newspaper
-from datetime import datetime, timezone
-from urllib.parse import urlparse
-
 from search_gov_crawler.elasticsearch.parse_html_scrapy import convert_html_scrapy
 from search_gov_crawler.search_gov_spiders.helpers import content
+from search_gov_crawler.search_gov_spiders.helpers import encoding
+from search_gov_crawler.elasticsearch.i14y_helper import ALLOWED_LANGUAGE_CODE, null_date, \
+    get_url_path, get_base_extension, current_utc_iso, generate_url_sha256, \
+    get_domain_name, detect_lang, summarize_text
 
-# fmt: off
-ALLOWED_LANGUAGE_CODE = (
-    "ar", "bg", "bn", "ca", "cs", "da", "de", "el", "en", "es", "et", "fa", "fr",
-    "he", "hi", "hr", "ht", "hu", "hy", "id", "it", "ja", "km", "ko", "lt", "lv",
-    "mk", "nl", "pl", "ps", "pt", "ro", "ru", "sk", "so", "sq", "sr", "sw", "th",
-    "tr", "uk", "ur", "uz", "vi", "zh",
-)
-# fmt: on
 
-def checkDate(article_date):
-    if article_date != "": 
-        return article_date
-    else:
-        return None 
-    
-def convert_html(html_content: str, url: str):
+def convert_html(response_bytes: bytes, url: str, response_language: str = None):
     """Extracts and processes article content from HTML using newspaper4k."""
+    html_content = encoding.decode_http_response(response_bytes=response_bytes)
     config = newspaper.Config()
     config.fetch_images = False  # we are not using images, do not fetch!
     config.clean_article_html = False  # we are not using article_html, so don't clean it!
@@ -43,6 +26,7 @@ def convert_html(html_content: str, url: str):
 
     title = article.title or article.meta_site_name or article_backup["title"] or None
     description = article.meta_description or article.summary or article_backup["description"] or None
+    tags = article.tags or article.keywords or article.meta_keywords or article_backup["keywords"] or None
 
     time_now_str = current_utc_iso()
     path = article.url or article_backup["url"] or url
@@ -50,29 +34,34 @@ def convert_html(html_content: str, url: str):
     basename, extension = get_base_extension(url)
     sha_id = generate_url_sha256(path)
 
-    language = article.meta_lang or article_backup["language"]
+    language = article.meta_lang or article_backup["language"] or response_language or detect_lang(main_content)
+    language = language[:2] if language else None
     valid_language = f"_{language}" if language in ALLOWED_LANGUAGE_CODE else ""
 
+    summary, keywords = summarize_text(text=main_content, lang_code=language)
+    tags = tags or keywords
+    description = description or summary
+
     return {
         "audience": article_backup["audience"],
-        "changed": checkDate(article_backup["changed"]),
+        "changed": null_date(article_backup["changed"]),
         "click_count": None,
         "content_type": "article",
-        "created_at": checkDate(article_backup["created_at"]) or time_now_str,
+        "created_at": null_date(article_backup["created_at"]) or time_now_str,
         "created": None,
         "_id": sha_id,
         "id": sha_id,
         "thumbnail_url": article.meta_img or article.top_image or article_backup["thumbnail_url"] or None,
-        "language": article.meta_lang,
+        "language": language,
         "mime_type": "text/html",
         "path": path,
         "promote": None,
         "searchgov_custom1": None,
         "searchgov_custom2": None,
         "searchgov_custom3": None,
-        "tags": article.tags or article.keywords or article.meta_keywords or article_backup["keywords"],
+        "tags": tags,
         "updated_at": time_now_str,
-        "updated": checkDate(article.publish_date) or checkDate(article_backup["created_at"]),
+        "updated": null_date(article.publish_date) or null_date(article_backup["created_at"]),
         f"title{valid_language}": title,
         f"description{valid_language}": content.sanitize_text(description),
         f"content{valid_language}": content.sanitize_text(main_content),
@@ -81,37 +70,3 @@ def convert_html(html_content: str, url: str):
         "url_path": get_url_path(url),
         "domain_name": get_domain_name(url),
     }
-
-
-def ensure_http_prefix(url: str):
-    return url if url.startswith(("http://", "https://")) else f"https://{url}"
-
-def get_url_path(url: str) -> str:
-    """Extracts the path from a URL."""
-    url = ensure_http_prefix(url)
-    return urlparse(url).path
-
-
-def get_base_extension(url: str) -> tuple[str, str]:
-    """Extracts the basename and file extension from a URL."""
-    url = ensure_http_prefix(url)
-    basename, extension = os.path.splitext(os.path.basename(urlparse(url).path))
-    return basename, extension
-
-
-def current_utc_iso() -> str:
-    """Returns the current UTC timestamp in ISO format."""
-    return datetime.now(tz=UTC).isoformat(timespec="milliseconds") + "Z"
-
-
-def generate_url_sha256(url: str) -> str:
-    """Generates a SHA-256 hash for a given URL."""
-    url = ensure_http_prefix(url)
-    return hashlib.sha256(url.encode()).hexdigest()
-
-
-def get_domain_name(url: str) -> str:
-    """Extracts the domain from a URL, support www (only if the url was parsed with it) ensuring consistency."""
-    url = ensure_http_prefix(url)
-    parsed = urlparse(url)
-    return parsed.netloc
diff --git a/search_gov_crawler/elasticsearch/convert_pdf_i14y.py b/search_gov_crawler/elasticsearch/convert_pdf_i14y.py
new file mode 100644
index 00000000..f5004fbc
--- /dev/null
+++ b/search_gov_crawler/elasticsearch/convert_pdf_i14y.py
@@ -0,0 +1,202 @@
+import re
+from io import BytesIO
+from pypdf import PdfReader
+from datetime import UTC, datetime, timedelta
+from search_gov_crawler.search_gov_spiders.helpers import content
+from search_gov_crawler.elasticsearch.i14y_helper import ALLOWED_LANGUAGE_CODE, null_date, \
+    get_url_path, get_base_extension, current_utc_iso, generate_url_sha256, get_domain_name, \
+    summarize_text, separate_file_name, detect_lang
+
+def add_title_and_filename(key: str, title_key: str, doc: dict):
+    """
+    Adds PDF's title and file name to the provided key.
+    Used mainly to improve index and relevance.
+
+    Args:
+        key: str The key to use to apply the change, eg "content"
+        doc: dict The i14y document the changes will be applied to
+
+    Returns:
+        None The changes are applied to the document as a referance/pointer
+    """
+    doc[key] = f"{doc[title_key]} {doc["basename"]}.{doc["extension"]} {doc[key]}"
+
+def get_links_set(reader: PdfReader):
+    """
+    Returns a set of links for all pages in the PDF
+
+    Args:
+        reader: PdfReader from pypdf
+
+    Returns:
+        (list[str]) unique set of links
+    """
+    key = "/Annots"
+    uri = "/URI"
+    ank = "/A"
+    links = {}
+
+    for page in reader.pages:
+        text = page.extract_text()
+        page_links = re.findall(r"https?://\S+|www\.\S+", text)
+        # Get all visible links
+        for link in page_links:
+            links[link] = link
+        
+        # Get all hidden links
+        page_object = page.get_object()
+        if key in page_object.keys():
+            ann = page_object[key]
+            for a in ann:
+                u = a.get_object()
+                if uri in u[ank].keys():
+                    link = u[ank][uri]
+                    links[link] = link
+    return links.values()
+
+def convert_pdf(response_bytes: bytes, url: str, response_language: str = None):
+    """Extracts and processes PDF content using pypdf."""
+    pdf_stream = BytesIO(response_bytes)
+    reader = PdfReader(pdf_stream)
+
+    if reader.is_encrypted:
+        return None
+
+    meta_values = get_pdf_meta(reader)
+    
+    basename, extension = get_base_extension(url)
+    title = meta_values.get("Title") or separate_file_name(f"{basename}.{extension}")
+    main_content = get_pdf_text(reader) or title
+
+    sha_id = generate_url_sha256(url)
+
+    language = meta_values.get("Lang") or response_language or detect_lang(main_content)
+    language = language[:2] if language else None
+    valid_language = f"_{language}" if language in ALLOWED_LANGUAGE_CODE else ""
+    
+    description, keywords = summarize_text(text=main_content, lang_code=language)
+
+    time_now_str = current_utc_iso()
+
+    content_key = f"content{valid_language}"
+    description_key = f"description{valid_language}"
+    title_key = f"title{valid_language}"
+
+    i14y_doc = {
+        "audience": None,
+        "changed": null_date(meta_values.get("ModDate") or meta_values.get("SourceModified")),
+        "click_count": None,
+        "content_type": None,
+        "created_at": null_date(meta_values.get("CreationDate")) or time_now_str,
+        "created": None,
+        "_id": sha_id,
+        "id": sha_id,
+        "thumbnail_url": None,
+        "language": language,
+        "mime_type": "application/pdf",
+        "path": url,
+        "promote": None,
+        "searchgov_custom1": None,
+        "searchgov_custom2": None,
+        "searchgov_custom3": None,
+        "tags": keywords,
+        "updated_at": time_now_str,
+        "updated": null_date(meta_values.get("CreationDate")),
+        title_key: title,
+        description_key: content.sanitize_text(description),
+        content_key: content.sanitize_text(main_content),
+        "basename": basename,
+        "extension": extension or None,
+        "url_path": get_url_path(url),
+        "domain_name": get_domain_name(url),
+    }
+
+    add_title_and_filename(content_key, title_key, i14y_doc)
+    add_title_and_filename(description_key, title_key, i14y_doc)
+    all_links = get_links_set(reader)
+    i14y_doc[content_key] = f"{i14y_doc[content_key]} {" ".join(all_links) if len(all_links) > 0 else ""}"
+
+    return i14y_doc
+    
+
+def get_pdf_text(reader: PdfReader) -> str:
+    """
+    Returns clean text/content from all pdf pages
+
+    Args:
+        reader: PdfReader from pypdf
+
+    Returns:
+        (string) without new any special characters
+    """
+    text = ""
+    for page in reader.pages:
+        text += page.extract_text() + " "
+    return text
+
+
+def get_pdf_meta(reader: PdfReader):
+    """
+    Returns pdf metadata
+
+    Args:
+        reader: PdfReader from pypdf
+
+    Returns:
+        metadata object with possible keys: https://exiftool.org/TagNames/PDF.html
+    """
+    clean_metadata = {}
+    metadata: dict = dict(reader.metadata)
+    if isinstance(metadata, dict):
+        for key, value in metadata.items():
+            new_key = None
+            if key.startswith("/"):
+                new_key = key[1:]
+            new_key = new_key or key
+            clean_metadata[new_key] = parse_if_date(value)
+    return clean_metadata
+
+
+def parse_if_date(value, apply_tz_offset: bool = False):
+    """
+    Parses a value as date if matched the conventional pdf/exif date format. If parsing fails,
+    returns the original value
+
+    Args:
+        value: The value to parse.
+
+    Returns:
+        A datetime.datetime object if parsing is successful, otherwise the original value.
+    """
+    if not isinstance(value, str) or not value.startswith("D:"):
+        return content.sanitize_text(value)
+
+    date_string = value[2:] # Remove the "D:" prefix
+
+    """
+    Example of matched date values:
+        "D:20191018122555-04'00'"
+        "D:20191018162538"
+    """
+    match = re.match(r"(\d{4})(\d{2})(\d{2})(\d{2})?(\d{2})?(\d{2})?([+-]\d{2})?'?(\d{2})?'?", date_string)
+
+    if match:
+        year = int(match.group(1))
+        month = int(match.group(2))
+        day = int(match.group(3))
+        hour = int(match.group(4)) if match.group(4) else 0
+        minute = int(match.group(5)) if match.group(5) else 0
+        second = int(match.group(6)) if match.group(6) else 0
+        tz_hour = int(match.group(7)[:3]) if match.group(7) else 0
+        tz_minute = int(match.group(7)[4:]) if match.group(7) and len(match.group(7)) > 4 else 0
+
+        try:
+            dt = datetime(year, month, day, hour, minute, second)
+            if match.group(7) and apply_tz_offset: # handle timezone offset if matched
+                tz_sign = 1 if tz_hour >= 0 else -1
+                tz_offset = timedelta(hours=tz_hour, minutes=tz_minute)
+                dt = dt - tz_sign * tz_offset
+            return dt
+        except ValueError as err:
+            raise f"Failed to parse Date value \"{value}\":\n{str(err)}"
+    return content.sanitize_text(value)
diff --git a/search_gov_crawler/elasticsearch/es_batch_upload.py b/search_gov_crawler/elasticsearch/es_batch_upload.py
index f8fcef6d..c8cf1dbf 100644
--- a/search_gov_crawler/elasticsearch/es_batch_upload.py
+++ b/search_gov_crawler/elasticsearch/es_batch_upload.py
@@ -10,6 +10,7 @@
 from scrapy.spiders import Spider
 
 from search_gov_crawler.elasticsearch.convert_html_i14y import convert_html
+from search_gov_crawler.elasticsearch.convert_pdf_i14y import convert_pdf
 from search_gov_crawler.search_gov_spiders.extensions.json_logging import LOG_FMT
 
 # limit excess INFO messages from elasticsearch that are not tied to a spider
@@ -33,11 +34,16 @@ def __init__(self, batch_size: int = 50):
         self._env_es_password = os.environ.get("ES_PASSWORD", "")
         self._executor = ThreadPoolExecutor(max_workers=5)  # Reuse one executor
 
-    def add_to_batch(self, html_content: str, url: str, spider: Spider):
+    def add_to_batch(self, response_bytes: bytes, url: str, spider: Spider, response_language: str, content_type: str):
         """
         Add a document to the batch for Elasticsearch upload.
         """
-        doc = convert_html(html_content=html_content, url=url)
+        doc = None
+        if content_type == "text/html":
+            doc = convert_html(response_bytes=response_bytes, url=url, response_language=response_language)
+        elif content_type == "application/pdf":
+            doc = convert_pdf(response_bytes=response_bytes, url=url, response_language=response_language)
+
         if doc:
             self._current_batch.append(doc)
 
diff --git a/search_gov_crawler/elasticsearch/i14y_helper.py b/search_gov_crawler/elasticsearch/i14y_helper.py
new file mode 100644
index 00000000..c3afb335
--- /dev/null
+++ b/search_gov_crawler/elasticsearch/i14y_helper.py
@@ -0,0 +1,141 @@
+import re
+import os
+import hashlib
+from langdetect import detect
+from nltk.corpus import stopwords
+from nltk.tokenize import word_tokenize, sent_tokenize
+from datetime import UTC, datetime
+from urllib.parse import urlparse
+
+
+# fmt: off
+ALLOWED_LANGUAGE_CODE = {
+    "ar": "arabic", "bg": "bulgarian", "bn": "bengali", "ca": "catalan", "cs": "czech", "da": "danish",
+    "de": "german", "el": "greek", "en": "english", "es": "spanish", "et": "estonian", "fa": "persian", "fr": "french",
+    "he": "hebrew", "hi": "hindi", "hr": "croatian", "ht": "haitian creole", "hu": "hungarian", "hy": "armenian", "id": "indonesian",
+    "it": "italian", "ja": "japanese", "km": "khmer", "ko": "korean", "lt": "lithuanian", "lv": "latvian", "mk": "macedonian",
+    "nl": "dutch", "pl": "polish", "ps": "pashto", "pt": "portuguese", "ro": "romanian", "ru": "russian", "sk": "slovak", "so": "somali",
+    "sq": "albanian", "sr": "serbian", "sw": "swahili", "th": "thai", "tr": "turkish", "uk": "ukrainian", "ur": "urdu", "uz": "uzbek",
+    "vi": "vietnamese", "zh": "chinese",
+}
+# fmt: on
+
+def null_date(article_date):
+    """
+    Convert falsey date values (like an empty string) to None,
+    which will yield a null value in elasticsearch
+
+    Args:
+        article_date (str | Any): Date value that needs to fallback to null
+
+    Returns:
+        str: Either the original date, or NoneValue
+    """
+    return article_date or None
+
+def detect_lang(text: str) -> str:
+    """
+    Detect language based on charachters and encoding
+
+    Args:
+        text (str): The text with the language in question
+
+    Returns:
+        str: A two letter language code, eg: "en", "es", "zh", ...
+    """
+    try:
+        lang = detect(text[:64])
+        return lang[:2] if len(lang) > 1 else None
+    except Exception:
+        pass
+    return None
+    
+
+def summarize_text(text: str, lang_code: str=None):
+    """
+    Summarizes text and extracts keywords using nltk, and calculates execution time.
+
+    Args:
+        text (str): extracted text/content
+
+    Returns:
+        tuple: (summary, keywords)
+    """
+
+    if (lang_code not in ALLOWED_LANGUAGE_CODE) or not text or type(text) != str:
+        return None, None
+    
+    stop_words = set(stopwords.words(ALLOWED_LANGUAGE_CODE[lang_code]))
+    sentences = sent_tokenize(text)
+
+    word_frequencies = {}
+    sentence_scores = {}
+
+    for sentence in sentences:
+        for word in word_tokenize(sentence.lower()):
+            if word.isalnum() and word not in stop_words:
+                if word not in word_frequencies:
+                    word_frequencies[word] = 1
+                else:
+                    word_frequencies[word] += 1
+
+            if word in word_frequencies:
+                if sentence not in sentence_scores:
+                    sentence_scores[sentence] = word_frequencies[word]
+                else:
+                    sentence_scores[sentence] += word_frequencies[word]
+
+    summary_sentences = sorted(sentence_scores, key=sentence_scores.get, reverse=True)[:3]
+    summary = ' '.join(summary_sentences)
+
+    sorted_words = sorted(word_frequencies, key=word_frequencies.get, reverse=True)[:10]
+    keywords = ", ".join(sorted_words)
+
+    return summary, keywords
+
+
+def separate_file_name(file_name):
+    """
+    Separates a file name into words, maintaining capitalization.
+    """
+    base_name = file_name.rsplit(".", 1)[0].replace(".", " ")
+    words = re.split(r"(?<!^)(?=[A-Z][a-z])|(?<=[a-z])(?=[A-Z])|[-_+~,%]|(?<=\D)(?=\d)|(?<=\d)(?=\D)", base_name)
+    return " ".join(words)
+
+
+def ensure_http_prefix(url: str):
+    """Add http prefix if missing."""
+    return url if url.startswith(("http://", "https://")) else f"https://{url}"
+
+
+def get_url_path(url: str) -> str:
+    """Extracts the path from a URL."""
+    url = ensure_http_prefix(url)
+    return urlparse(url).path
+
+
+def get_base_extension(url: str) -> tuple[str, str]:
+    """Extracts the basename and file extension from a URL."""
+    url = ensure_http_prefix(url)
+    basename, extension = os.path.splitext(os.path.basename(urlparse(url).path))
+    if extension.startswith("."):
+        extension = extension[1:]
+    return basename, extension
+
+
+def current_utc_iso() -> str:
+    """Returns the current UTC timestamp in ISO format."""
+    return datetime.now(tz=UTC).isoformat(timespec="milliseconds") + "Z"
+
+
+def generate_url_sha256(url: str) -> str:
+    """Generates a SHA-256 hash for a given URL."""
+    url = ensure_http_prefix(url)
+    return hashlib.sha256(url.encode()).hexdigest()
+
+
+def get_domain_name(url: str) -> str:
+    """Extracts the domain from a URL, support www (only if the url was parsed with it) ensuring consistency."""
+    url = ensure_http_prefix(url)
+    parsed = urlparse(url)
+    return parsed.netloc
diff --git a/search_gov_crawler/elasticsearch/install_nltk.py b/search_gov_crawler/elasticsearch/install_nltk.py
index c7ef1d53..12df90be 100644
--- a/search_gov_crawler/elasticsearch/install_nltk.py
+++ b/search_gov_crawler/elasticsearch/install_nltk.py
@@ -9,7 +9,7 @@ def download_nltk_package(package):
     except Exception as e:
         return f"Failed to download {package}: {str(e)}"
 
-nltk_packages = ["popular", "punkt_tab", "punkt", "perluniprops", "nonbreaking_prefixes"]
+nltk_packages = ["popular", "punkt_tab", "punkt", "perluniprops", "nonbreaking_prefixes", "stopwords"]
 
 with ThreadPoolExecutor(max_workers=len(nltk_packages)) as executor:
     future_to_package = {
diff --git a/search_gov_crawler/requirements.txt b/search_gov_crawler/requirements.txt
index 9f9c73fd..a7fd34de 100644
--- a/search_gov_crawler/requirements.txt
+++ b/search_gov_crawler/requirements.txt
@@ -23,3 +23,6 @@ newspaper4k[all]==0.9.3.1 # This also installs the nltk library
 cchardet==2.2.0a2 # C version of chardet
 pytest-asyncio==0.25.3
 python-dotenv==1.0.1
+
+pypdf==5.3.1
+langdetect==1.0.9
diff --git a/search_gov_crawler/search_gov_spiders/helpers/domain_spider.py b/search_gov_crawler/search_gov_spiders/helpers/domain_spider.py
index 1a3c4eb7..5e5e5d0b 100755
--- a/search_gov_crawler/search_gov_spiders/helpers/domain_spider.py
+++ b/search_gov_crawler/search_gov_spiders/helpers/domain_spider.py
@@ -4,6 +4,7 @@
 from typing import Any, Optional
 
 from scrapy.linkextractors import LinkExtractor
+from scrapy.http.response import Response
 
 # fmt: off
 FILTER_EXTENSIONS = [
@@ -39,6 +40,7 @@
 
 ES_ALLOWED_CONTENT_TYPE = [
     "text/html",
+    "application/pdf",
 ]
 
 ALLOWED_CONTENT_TYPE_OUTPUT_MAP = {
@@ -72,14 +74,26 @@ def split_allowed_domains(allowed_domains: str) -> list[str]:
     return host_only_domains
 
 
-def is_valid_content_type(content_type_header: Any, output_target: str) -> bool:
+def is_valid_content_type(content_type_header: str, output_target: str) -> bool:
     """Check that content type header is in list of allowed values"""
+    if not content_type_header:
+        return None
     content_type_header = str(content_type_header)
     for type_regex in ALLOWED_CONTENT_TYPE_OUTPUT_MAP[output_target]:
         if re.search(type_regex, content_type_header):
             return True
     return False
 
+def get_simple_content_type(content_type_header: str, output_target: str) -> str:
+    """Returns simple content time like: \"text/html\""""
+    if not content_type_header:
+        return None
+    content_type_header = str(content_type_header)
+    for type_regex in ALLOWED_CONTENT_TYPE_OUTPUT_MAP[output_target]:
+        if re.search(type_regex, content_type_header):
+            return type_regex
+    return None
+
 
 def get_crawl_sites(crawl_file_path: Optional[str] = None) -> list[dict]:
     """Read in list of crawl sites from json file"""
@@ -130,3 +144,24 @@ def validate_spider_arguments(allowed_domains: str | None, start_urls: str | Non
             f"{list(ALLOWED_CONTENT_TYPE_OUTPUT_MAP.keys())}"
         )
         raise ValueError(msg)
+
+
+def get_response_language_code(response: Response) -> str:
+    """
+    Retrieves the two-letter language code from Content-Language header of a response.
+
+    Args:
+        response (Response): Scrapy's response object
+
+    Returns:
+        str: The two-letter language code, or None if not found or invalid.
+    """
+    try:
+        header_name = "Content-Language"
+        content_language = response.headers.get(header_name, response.headers.get(header_name.lower(), None))
+        if content_language:
+            return content_language[:2]
+    except Exception:
+        pass
+    return None
+    
diff --git a/search_gov_crawler/search_gov_spiders/items.py b/search_gov_crawler/search_gov_spiders/items.py
index ce1f3eaf..bbf5faad 100755
--- a/search_gov_crawler/search_gov_spiders/items.py
+++ b/search_gov_crawler/search_gov_spiders/items.py
@@ -7,6 +7,8 @@
 
 class SearchGovSpidersItem(scrapy.Item):
     """Class for Item which is a container for every returned scraped page"""
-    html_content = scrapy.Field()
+    response_bytes = scrapy.Field()
     url = scrapy.Field()
     output_target = scrapy.Field()
+    response_language = scrapy.Field()
+    content_type = scrapy.Field()
diff --git a/search_gov_crawler/search_gov_spiders/pipelines.py b/search_gov_crawler/search_gov_spiders/pipelines.py
index 206f7e0b..9b04c00d 100755
--- a/search_gov_crawler/search_gov_spiders/pipelines.py
+++ b/search_gov_crawler/search_gov_spiders/pipelines.py
@@ -62,7 +62,9 @@ def process_item(self, item: SearchGovSpidersItem, spider: Spider) -> SearchGovS
             self._process_file_item(url)
         
         safe_del(item, "output_target")
-        safe_del(item, "html_content")
+        safe_del(item, "response_bytes")
+        safe_del(item, "response_language")
+        safe_del(item, "content_type")
 
         return item
     
@@ -74,13 +76,18 @@ def _get_elasticsearch_client(self) -> SearchGovElasticsearch:
     
     def _process_es_item(self, item: SearchGovSpidersItem, spider: Spider):
         url = item.get("url", None)
-        html_content = item.get("html_content", None)
-
-        if not html_content:
-            spider.logger.error(f"Missing 'html_content' for url: {url}")
-            raise DropItem("Missing URL or HTML in item")
+        response_bytes = item.get("response_bytes", None)
+        response_language = item.get("response_language", None)
+        content_type = item.get("content_type", None)
+
+        if not response_bytes:
+            err = f"Missing 'response_bytes' for url: {url}"
+            spider.logger.error(err)
+            raise DropItem(err)
         try:
-            self._get_elasticsearch_client().add_to_batch(html_content=html_content, url=url, spider=spider)
+            self._get_elasticsearch_client().add_to_batch(
+                response_bytes=response_bytes, url=url, spider=spider, response_language=response_language, content_type=content_type
+            )
         except Exception as e:
             raise DropItem(f"Item 'elasticsearch' add_to_batch() failed: {str(e)}")
 
diff --git a/search_gov_crawler/search_gov_spiders/spiders/domain_spider.py b/search_gov_crawler/search_gov_spiders/spiders/domain_spider.py
index 8f9513fe..92e344f4 100755
--- a/search_gov_crawler/search_gov_spiders/spiders/domain_spider.py
+++ b/search_gov_crawler/search_gov_spiders/spiders/domain_spider.py
@@ -98,7 +98,12 @@ def parse_item(self, response: Response):
         @returns items 1 1
         @scrapes url
         """
-
-        if helpers.is_valid_content_type(response.headers.get("content-type", None), output_target=self.output_target):
-            html_content = encoding.decode_http_response(response_bytes=response.body)
-            yield SearchGovSpidersItem(url=response.url, html_content=html_content, output_target=self.output_target)
+        content_type_name = "Content-Type"
+        content_type_value = response.headers.get(content_type_name, response.headers.get(content_type_name.lower(), None))
+        if helpers.is_valid_content_type(content_type_value, output_target=self.output_target):
+            yield SearchGovSpidersItem(
+                url=response.url,
+                response_bytes=response.body,
+                output_target=self.output_target,
+                content_type=helpers.get_simple_content_type(content_type_value, output_target=self.output_target)
+            )
diff --git a/search_gov_crawler/search_gov_spiders/spiders/domain_spider_js.py b/search_gov_crawler/search_gov_spiders/spiders/domain_spider_js.py
index 2f632cf4..b6cadee2 100755
--- a/search_gov_crawler/search_gov_spiders/spiders/domain_spider_js.py
+++ b/search_gov_crawler/search_gov_spiders/spiders/domain_spider_js.py
@@ -131,9 +131,16 @@ def parse_item(self, response: Response):
         @scrapes url
         """
 
-        if helpers.is_valid_content_type(response.headers.get("content-type", None), output_target=self.output_target):
-            html_content = encoding.decode_http_response(response_bytes=response.body)
-            yield SearchGovSpidersItem(url=response.url, html_content=html_content, output_target=self.output_target)
+        content_type_name = "Content-Type"
+        content_type_value = response.headers.get(content_type_name, response.headers.get(content_type_name.lower(), None))
+        if helpers.is_valid_content_type(content_type_value, output_target=self.output_target):
+            yield SearchGovSpidersItem(
+                url=response.url,
+                response_bytes=response.body,
+                output_target=self.output_target,
+                response_language=helpers.get_response_language_code(response),
+                content_type=helpers.get_simple_content_type(content_type_value, output_target=self.output_target)
+            )
 
     def set_playwright_usage(self, request: Request, _response: Response) -> Request:
         """Set meta tags for playwright to run"""
diff --git a/tests/search_gov_spiders/test_elasticsearch.py b/tests/search_gov_spiders/test_elasticsearch.py
index 6cfccf74..df5acc1c 100644
--- a/tests/search_gov_spiders/test_elasticsearch.py
+++ b/tests/search_gov_spiders/test_elasticsearch.py
@@ -21,6 +21,7 @@
     </html>
 """
 
+response_bytes = html_content.encode()
 
 @pytest.fixture
 def sample_spider():
@@ -81,10 +82,10 @@ async def test_add_to_batch(mock_convert_html, sample_spider):
     es_uploader = SearchGovElasticsearch(batch_size=2)
     mock_convert_html.return_value = {"_id": "1", "title": "Test Document"}
 
-    es_uploader.add_to_batch(html_content, "http://example.com/1", sample_spider)
+    es_uploader.add_to_batch(response_bytes, "http://example.com/1", sample_spider, "en", "text/html")
     assert len(es_uploader._current_batch) == 1
 
-    es_uploader.add_to_batch(html_content, "http://example.com/2", sample_spider)
+    es_uploader.add_to_batch(response_bytes, "http://example.com/2", sample_spider, "en", "text/html")
     assert len(es_uploader._current_batch) == 0
 
 
@@ -122,7 +123,7 @@ def test_add_to_batch_no_doc(mock_convert_html, sample_spider):
     es_uploader = SearchGovElasticsearch(batch_size=2)
     mock_convert_html.return_value = None
 
-    es_uploader.add_to_batch("<html></html>", "http://example.com/1", sample_spider)
+    es_uploader.add_to_batch(b"<html></html>", "http://example.com/1", sample_spider, "en", "text/html")
     assert len(es_uploader._current_batch) == 0
 
 
diff --git a/tests/search_gov_spiders/test_full_crawl.py b/tests/search_gov_spiders/test_full_crawl.py
index 62be3547..9c14761f 100755
--- a/tests/search_gov_spiders/test_full_crawl.py
+++ b/tests/search_gov_spiders/test_full_crawl.py
@@ -122,6 +122,7 @@ def mock_init(pipeline_cls, *_args, temp_dir=temp_dir, **_kwargs):
             pipeline_cls.current_file = open(pipeline_cls.file_path, "w", encoding="utf-8")
             pipeline_cls.file_open = False
             pipeline_cls._es = None
+            pipeline_cls.urls_batch = []
 
         monkeypatch.setattr(
             "search_gov_crawler.search_gov_spiders.pipelines.SearchGovSpidersPipeline.__init__", mock_init
diff --git a/tests/search_gov_spiders/test_i14y_helper.py b/tests/search_gov_spiders/test_i14y_helper.py
new file mode 100644
index 00000000..607d5fa7
--- /dev/null
+++ b/tests/search_gov_spiders/test_i14y_helper.py
@@ -0,0 +1,70 @@
+from search_gov_crawler.elasticsearch.i14y_helper import null_date, detect_lang, separate_file_name
+
+# Tests for null_date
+def test_null_date_valid_date():
+    assert null_date("2025-03-13") == "2025-03-13"
+
+def test_null_date_empty_string():
+    assert null_date("") is None
+
+def test_null_date_none():
+    assert null_date(None) is None
+
+def test_null_date_zero():
+    assert null_date(0) is None
+
+def test_null_date_false():
+    assert null_date(False) is None
+
+# Tests for detect_lang
+def test_detect_lang_english():
+    assert detect_lang("This is a test sentence in English. And, this is another test sentence in English.") == "en"
+
+def test_detect_lang_spanish():
+    assert detect_lang("Esta es una frase de prueba en español.") == "es"
+
+def test_detect_lang_chinese():
+    assert detect_lang("这是一个中文测试句子。") == "zh" or detect_lang("这是一个中文测试句子.") == "zh" or detect_lang("这是一个中文测试句子.") == "zh"
+
+def test_detect_lang_short_text():
+    assert detect_lang("How are you") == "en"
+
+def test_detect_lang_empty_text():
+    assert detect_lang("") is None
+
+def test_detect_lang_unrecognizable_text():
+    assert detect_lang("1234567890!@#$%^&*()") is None
+
+# Tests for separate_file_name
+def test_separate_file_name_camel_case():
+    assert separate_file_name("camelCaseFile.pdf") == "camel Case File"
+
+def test_separate_file_name_snake_case():
+    assert separate_file_name("snake_case_file.pdf") == "snake case file"
+
+def test_separate_file_name_kebab_case():
+    assert separate_file_name("kebab-case-file.pdf") == "kebab case file"
+
+def test_separate_file_name_pascal_case():
+    assert separate_file_name("PascalCaseFile.pdf") == "Pascal Case File"
+
+def test_separate_file_name_mixed_case():
+    assert separate_file_name("mixedCase123File.pdf") == "mixed Case 123 File"
+
+def test_separate_file_name_numbers():
+    assert separate_file_name("file123Test.pdf") == "file 123 Test"
+
+def test_separate_file_name_symbols():
+    assert separate_file_name("file_with-symbols+test,file~name%test.pdf") == "file with symbols test file name test"
+
+def test_separate_file_name_no_extension():
+    assert separate_file_name("noExtensionFile") == "no Extension File"
+
+def test_separate_file_name_multiple_dots():
+    assert separate_file_name("file.with.multiple.dots.pdf") == "file with multiple dots"
+
+def test_separate_file_name_empty():
+    assert separate_file_name("") == ""
+
+def test_separate_file_name_only_extension():
+    assert separate_file_name(".pdf") == ""
diff --git a/tests/search_gov_spiders/test_i14y.py b/tests/search_gov_spiders/test_i14y_html.py
similarity index 86%
rename from tests/search_gov_spiders/test_i14y.py
rename to tests/search_gov_spiders/test_i14y_html.py
index 9bafd389..3bad9617 100644
--- a/tests/search_gov_spiders/test_i14y.py
+++ b/tests/search_gov_spiders/test_i14y_html.py
@@ -1,4 +1,4 @@
-from search_gov_crawler.elasticsearch import convert_html_i14y as conversion
+from search_gov_crawler.elasticsearch import convert_html_i14y
 from search_gov_crawler.search_gov_spiders.helpers import content
 
 def test_convert_html_valid_article():
@@ -17,8 +17,9 @@ def test_convert_html_valid_article():
     </body>
     </html>
     """
+    response_bytes = html_content.encode()
     url = "https://example.com/test-article"
-    result = conversion.convert_html(html_content, url)
+    result = convert_html_i14y.convert_html(response_bytes, url, "en")
 
     assert result is not None
     assert result["title_en"] == "Test Article Title"
@@ -44,7 +45,7 @@ def test_convert_html_no_content():
     </html>
     """
     url = "https://example.com/test-article"
-    result = conversion.convert_html(html_content, url)
+    result = convert_html_i14y.convert_html(html_content.encode(), url, "en")
     assert result is None
 
 def test_convert_html_no_title_or_description():
@@ -58,10 +59,11 @@ def test_convert_html_no_title_or_description():
     </html>
     """
     url = "https://example.com/test-article"
-    result = conversion.convert_html(html_content, url)
+    result = convert_html_i14y.convert_html(html_content.encode(), url, "en")
+    content = "This is the main content of the test article."
     assert result is not None
     assert result["title_en"] is None
-    assert result["description_en"] is None
+    assert result["description_en"] in content
     assert "This is the main content of the test article." in result["content_en"]
 
 def test_convert_html_with_meta_site_name():
@@ -77,7 +79,7 @@ def test_convert_html_with_meta_site_name():
     </html>
     """
     url = "https://example.com/test-article"
-    result = conversion.convert_html(html_content, url)
+    result = convert_html_i14y.convert_html(html_content.encode(), url, "en")
     assert result is not None
     assert result["title_en"] == "Example Site"  # Uses meta_site_name
     assert "This is the main content." in result["content_en"]
@@ -95,7 +97,7 @@ def test_convert_html_with_publish_date():
     </html>
     """
     url = "https://example.com/test-article"
-    result = conversion.convert_html(html_content, url)
+    result = convert_html_i14y.convert_html(html_content.encode(), url, "en")
     assert result is not None
     assert result["updated"] is not None # newspaper4k may or may not parse date from meta; this checks for any value.
 
@@ -112,7 +114,7 @@ def test_convert_html_with_out_publish_date():
     </html>
     """
     url = "https://example.com/test-article"
-    result = conversion.convert_html(html_content, url)
+    result = convert_html_i14y.convert_html(html_content.encode(), url, "en")
     assert result is not None
     assert result["updated"] is not ""
     assert result["updated"] is None # newspaper4k may or may not parse date from meta; this checks for any value.
@@ -136,7 +138,7 @@ def test_convert_html_languages():
     """
     url = "https://example.cn/article"
     
-    result = conversion.convert_html(html_content, url)
+    result = convert_html_i14y.convert_html(html_content.encode(), url, "zh")
 
     assert result is not None
     assert result["content_zh"] == "労化合測断秒化任面件気子人球分向無圧。了作果批入選教済球主運私信成笑論情禁。首着場研打表阪東日善能最囲値名陣必。想必愛交備見事新演内高青録断狙。詳期斉幕善確込対危継属会提円和動会分子。中常特処秘局創企真刊葉戸獲人師。前場明持二本聞通調写何観。薫大本設紋証済球取縮不園。辺案惑報湖買含応給奥専申琴真集情月続。"
diff --git a/tests/search_gov_spiders/test_i14y_pdf.py b/tests/search_gov_spiders/test_i14y_pdf.py
new file mode 100644
index 00000000..95cc9391
--- /dev/null
+++ b/tests/search_gov_spiders/test_i14y_pdf.py
@@ -0,0 +1,192 @@
+import pytest
+from datetime import datetime
+from unittest.mock import patch, MagicMock
+from search_gov_crawler.elasticsearch import convert_pdf_i14y
+
+class FakePage:
+    def __init__(self, text):
+        self._text = text
+        self._object = {}
+
+    def extract_text(self):
+        return self._text
+    
+    def get_object(self):
+        return self._object
+
+class FakePdfReader:
+    def __init__(self, stream, is_encrypted=False, pages=None, metadata=None):
+        self.is_encrypted = is_encrypted
+        self.pages = pages if pages is not None else []
+        self.metadata = metadata if metadata is not None else {}
+
+# Dummy helper functions to override external dependencies. We're already testing them in other places
+def dummy_get_base_extension(url):
+    return ("fake_basename", "pdf")
+
+def dummy_separate_file_name(filename):
+    return "Fake Title from filename"
+
+def dummy_summarize_text(text, lang_code):
+    # Return a tuple: (description, list of keywords)
+    return ("Fake description", ["keyword1", "keyword2"])
+
+def dummy_generate_url_sha256(url):
+    return "dummy_sha"
+
+def dummy_detect_lang(text):
+    return "en"
+
+def dummy_current_utc_iso():
+    return "2023-01-01T00:00:00Z"
+
+def dummy_null_date(date_str):
+    # For testing, simply return the date_str (or a default if none provided)
+    return date_str if date_str else "null_date"
+
+def dummy_get_url_path(url):
+    return "/fake/path"
+
+def dummy_get_domain_name(url):
+    return "fake.domain.com"
+
+def dummy_sanitize_text(text):
+    return text.strip()
+
+# ----- Pytest fixture to patch helper functions -----
+
+@pytest.fixture(autouse=True)
+def patch_helpers(monkeypatch):
+    monkeypatch.setattr(convert_pdf_i14y, "get_base_extension", dummy_get_base_extension)
+    monkeypatch.setattr(convert_pdf_i14y, "separate_file_name", dummy_separate_file_name)
+    monkeypatch.setattr(convert_pdf_i14y, "summarize_text", dummy_summarize_text)
+    monkeypatch.setattr(convert_pdf_i14y, "generate_url_sha256", dummy_generate_url_sha256)
+    monkeypatch.setattr(convert_pdf_i14y, "detect_lang", dummy_detect_lang)
+    monkeypatch.setattr(convert_pdf_i14y, "current_utc_iso", dummy_current_utc_iso)
+    monkeypatch.setattr(convert_pdf_i14y, "null_date", dummy_null_date)
+    monkeypatch.setattr(convert_pdf_i14y, "get_url_path", dummy_get_url_path)
+    monkeypatch.setattr(convert_pdf_i14y, "get_domain_name", dummy_get_domain_name)
+    monkeypatch.setattr(convert_pdf_i14y.content, "sanitize_text", dummy_sanitize_text)
+    # Ensure the allowed language list includes "en" so that we get a valid language suffix.
+    monkeypatch.setattr(convert_pdf_i14y, "ALLOWED_LANGUAGE_CODE", {"en": "english"})
+
+# ----- Tests for helper functions -----
+
+def test_get_pdf_text():
+    """Test that get_pdf_text concatenates text from each page."""
+    fake_page_content_1 = """
+    Lorem Ipsum is simply dummy text of the printing and typesetting industry.
+    Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, 
+    when an unknown printer took a galley of type and scrambled it to make a type specimen book. 
+    It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged.
+    """
+    fake_page_content_2 = f"Page 2 content: {fake_page_content_1}"
+    pages = [FakePage(fake_page_content_1), FakePage(fake_page_content_2)]
+    fake_reader = FakePdfReader(None, pages=pages)
+    result = convert_pdf_i14y.get_pdf_text(fake_reader)
+    expected = f"{fake_page_content_1} {fake_page_content_2} "
+    assert result == expected
+
+def test_get_pdf_meta():
+    """Test that metadata is cleaned and dates are parsed."""
+    metadata = {
+        "/Title": "Fake Title",
+        "/CreationDate": "D:20230101000000"
+    }
+    fake_reader = FakePdfReader(None, metadata=metadata)
+    meta = convert_pdf_i14y.get_pdf_meta(fake_reader)
+    # Check that the Title is preserved and the CreationDate is parsed to a datetime.
+    assert meta.get("Title") == "Fake Title"
+    creation_date = meta.get("CreationDate")
+    assert isinstance(creation_date, datetime)
+    assert creation_date == datetime(2023, 1, 1, 0, 0, 0)
+
+def test_parse_if_date_valid():
+    """Test parse_if_date with a valid PDF date string."""
+    date_str = "D:20230101023045"
+    result = convert_pdf_i14y.parse_if_date(date_str)
+    expected = datetime(2023, 1, 1, 2, 30, 45)
+    assert result == expected
+
+def test_parse_if_date_invalid():
+    """Test parse_if_date with an invalid date string starting with 'D:'."""
+    invalid_date = "D:invalid"
+    # Since the regex does not match, it should fall back to sanitizing the text.
+    result = convert_pdf_i14y.parse_if_date(invalid_date)
+    assert result == invalid_date.strip()
+
+def test_parse_if_date_non_date():
+    """Test parse_if_date with a non-string value (or a string not starting with 'D:')."""
+    non_date = "Not a date"
+    result = convert_pdf_i14y.parse_if_date(non_date)
+    assert result == non_date.strip()
+
+# ----- Tests for the main conversion function -----
+
+def test_convert_pdf_normal(monkeypatch):
+    """Test convert_pdf with a non-encrypted PDF simulation."""
+    fake_metadata = {
+        "/Title": "Fake Title",
+        "/CreationDate": "D:20230101000000"
+    }
+    pages = [FakePage("This is the content of the PDF.")]
+    fake_reader = FakePdfReader(None, is_encrypted=False, pages=pages, metadata=fake_metadata)
+    # Patch PdfReader so that any instantiation returns our fake_reader.
+    monkeypatch.setattr(convert_pdf_i14y, "PdfReader", lambda stream: fake_reader)
+    
+    response_bytes = b"dummy bytes representing pdf"
+    url = "http://example.com/fake.pdf"
+    result = convert_pdf_i14y.convert_pdf(response_bytes, url, response_language="en")
+    
+    # Check that the result is a dict with expected keys and values.
+    assert result is not None
+    # Since ALLOWED_LANGUAGE_CODE includes "en", we expect language suffix "_en" on these fields.
+    assert result["title_en"] == "Fake Title"
+    assert result["description_en"] == "Fake Title fake_basename.pdf Fake description"
+    assert result["content_en"] == "Fake Title fake_basename.pdf This is the content of the PDF. "
+    assert result["_id"] == "dummy_sha"
+    # Check values from dummy helpers.
+    assert result["basename"] == "fake_basename"
+    assert result["extension"] == "pdf"
+    assert result["url_path"] == "/fake/path"
+    assert result["domain_name"] == "fake.domain.com"
+
+def test_convert_pdf_encrypted(monkeypatch):
+    """Test convert_pdf when the PDF is encrypted (should return None)."""
+    fake_reader = FakePdfReader(None, is_encrypted=True)
+    monkeypatch.setattr(convert_pdf_i14y, "PdfReader", lambda stream: fake_reader)
+    response_bytes = b"dummy bytes representing pdf"
+    url = "http://example.com/encrypted.pdf"
+    result = convert_pdf_i14y.convert_pdf(response_bytes, url)
+    assert result is None
+
+
+def test_add_title_and_filename():
+    """Test that add_title_and_filename correctly formats the content."""
+    doc = {
+        "title_en": "Sample PDF",
+        "basename": "sample",
+        "extension": "pdf",
+        "content_en": "This is some sample content."
+    }
+    convert_pdf_i14y.add_title_and_filename("content_en", "title_en", doc)
+    expected_content = "Sample PDF sample.pdf This is some sample content."
+    assert doc["content_en"] == expected_content
+
+
+def test_get_links_set():
+    """Test that get_links_set extracts unique links from PDF pages."""
+    # Mocking PdfReader and its pages
+    fake_page1 = MagicMock()
+    fake_page1.extract_text.return_value = "Visit https://example.com for more info."
+    
+    fake_page2 = MagicMock()
+    fake_page2.extract_text.return_value = "Check out www.test.com and also https://example.com"
+    
+    fake_reader = MagicMock()
+    fake_reader.pages = [fake_page1, fake_page2]
+    
+    links = convert_pdf_i14y.get_links_set(fake_reader)
+    expected_links = {"https://example.com", "www.test.com"}
+    
+    assert set(links) == expected_links


Reviews:
user: login: selfdanielj
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T20:43:46Z
user: login: igoristic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T22:14:32Z
user: login: igoristic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T22:16:21Z
user: login: selfdanielj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T10:51:41Z
user: login: IsabelLaurenceau
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T11:43:02Z
user: login: igoristic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T15:13:51Z

Review comments:
diff_hunk: @@ -7,6 +7,52 @@
     get_url_path, get_base_extension, current_utc_iso, generate_url_sha256, get_domain_name, \
     summarize_text, separate_file_name, detect_lang
 
+def add_title_and_filename(key: str, title_key: str, doc: dict):
+    """
+    Adds PDF's title and file name to the provided key.
+    Used mainly to improve index and relevance.
+
+    Args:
+        key: str The key to use to apply the change, eg "content"
+        doc: dict The i14y document the changes will be applied to
+
+    Returns:
+        None The changes are applied to the document as a referance/pointer
+    """
+    doc[key] = f"{doc[title_key]} {doc["basename"]}.{doc["extension"]} {doc[key]}"
+
+def get_links_set(reader: PdfReader):
+    """
+    Returns a set of links for all pages in the PDF
+
+    Args:
+        reader: PdfReader from pypdf
+
+    Returns:
+        (list[str]) unique set of links
+    """
+    key = "/Annots"
+    uri = "/URI"
+    ank = "/A"
+    links = {}
+
+    for page in reader.pages:
+        text = page.extract_text()
+        page_links = re.findall(r"https?://\S+|www\.\S+", text)
+        # Get all visible links
+        for link in page_links:
+            links[link] = link
+        
+        # Get all hidden links
+        page_object = page.get_object()
+        if key in page_object.keys():
+            ann = page_object[key]
+            for a in ann:
+                u = a.get_object()
+                if uri in u[ank].keys():
+                    link = u[ank][uri]
+                    links[link] = link
+    return links.values()
path: search_gov_crawler/elasticsearch/convert_pdf_i14y.py
user: login: selfdanielj
body: Did you find this effective?  It seems overly complex just to add something back in that we stripped out.. did you look into if we could just not strip them out?
created_at: 2025-03-17T20:27:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -52,14 +102,21 @@ def convert_pdf(response_bytes: bytes, url: str, response_language: str = None):
         "tags": keywords,
         "updated_at": time_now_str,
         "updated": null_date(meta_values.get("CreationDate")),
-        f"title{valid_language}": title,
-        f"description{valid_language}": content.sanitize_text(description),
-        f"content{valid_language}": content.sanitize_text(main_content),
+        title_key: title,
+        description_key: content.sanitize_text(description),
+        content_key: content.sanitize_text(main_content),
         "basename": basename,
         "extension": extension or None,
         "url_path": get_url_path(url),
         "domain_name": get_domain_name(url),
     }
+
+    add_title_and_filename(content_key, title_key, i14y_doc)
+    add_title_and_filename(description_key, title_key, i14y_doc)
+    all_links = get_links_set(reader)
+    i14y_doc[content_key] = f"{i14y_doc[content_key]} {" ".join(all_links) if len(all_links) > 0 else ""}"
+
+    return i14y_doc
path: search_gov_crawler/elasticsearch/convert_pdf_i14y.py
user: login: selfdanielj
body: i don't particularly like hiding the dict manipulation in a function like this, at a glance i can't tell that those fields are getting modified.
created_at: 2025-03-17T20:28:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,6 +7,52 @@
     get_url_path, get_base_extension, current_utc_iso, generate_url_sha256, get_domain_name, \
     summarize_text, separate_file_name, detect_lang
 
+def add_title_and_filename(key: str, title_key: str, doc: dict):
+    """
+    Adds PDF's title and file name to the provided key.
+    Used mainly to improve index and relevance.
+
+    Args:
+        key: str The key to use to apply the change, eg "content"
+        doc: dict The i14y document the changes will be applied to
+
+    Returns:
+        None The changes are applied to the document as a referance/pointer
+    """
+    doc[key] = f"{doc[title_key]} {doc["basename"]}.{doc["extension"]} {doc[key]}"
+
+def get_links_set(reader: PdfReader):
+    """
+    Returns a set of links for all pages in the PDF
+
+    Args:
+        reader: PdfReader from pypdf
+
+    Returns:
+        (list[str]) unique set of links
+    """
+    key = "/Annots"
+    uri = "/URI"
+    ank = "/A"
+    links = {}
+
+    for page in reader.pages:
+        text = page.extract_text()
+        page_links = re.findall(r"https?://\S+|www\.\S+", text)
+        # Get all visible links
+        for link in page_links:
+            links[link] = link
+        
+        # Get all hidden links
+        page_object = page.get_object()
+        if key in page_object.keys():
+            ann = page_object[key]
+            for a in ann:
+                u = a.get_object()
+                if uri in u[ank].keys():
+                    link = u[ank][uri]
+                    links[link] = link
+    return links.values()
path: search_gov_crawler/elasticsearch/convert_pdf_i14y.py
user: login: igoristic
body: Yeah, I was trying to mimic the behavior in the original i14y. Not sure if this helps the indexing/etc. Though, I think it might be worth adding the "hidden" links
created_at: 2025-03-17T22:14:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -52,14 +102,21 @@ def convert_pdf(response_bytes: bytes, url: str, response_language: str = None):
         "tags": keywords,
         "updated_at": time_now_str,
         "updated": null_date(meta_values.get("CreationDate")),
-        f"title{valid_language}": title,
-        f"description{valid_language}": content.sanitize_text(description),
-        f"content{valid_language}": content.sanitize_text(main_content),
+        title_key: title,
+        description_key: content.sanitize_text(description),
+        content_key: content.sanitize_text(main_content),
         "basename": basename,
         "extension": extension or None,
         "url_path": get_url_path(url),
         "domain_name": get_domain_name(url),
     }
+
+    add_title_and_filename(content_key, title_key, i14y_doc)
+    add_title_and_filename(description_key, title_key, i14y_doc)
+    all_links = get_links_set(reader)
+    i14y_doc[content_key] = f"{i14y_doc[content_key]} {" ".join(all_links) if len(all_links) > 0 else ""}"
+
+    return i14y_doc
path: search_gov_crawler/elasticsearch/convert_pdf_i14y.py
user: login: igoristic
body: Not sure what you mean. This is normal functional pattern. Are there any examples/patterns we should try to follow?
created_at: 2025-03-17T22:16:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -52,14 +102,21 @@ def convert_pdf(response_bytes: bytes, url: str, response_language: str = None):
         "tags": keywords,
         "updated_at": time_now_str,
         "updated": null_date(meta_values.get("CreationDate")),
-        f"title{valid_language}": title,
-        f"description{valid_language}": content.sanitize_text(description),
-        f"content{valid_language}": content.sanitize_text(main_content),
+        title_key: title,
+        description_key: content.sanitize_text(description),
+        content_key: content.sanitize_text(main_content),
         "basename": basename,
         "extension": extension or None,
         "url_path": get_url_path(url),
         "domain_name": get_domain_name(url),
     }
+
+    add_title_and_filename(content_key, title_key, i14y_doc)
+    add_title_and_filename(description_key, title_key, i14y_doc)
+    all_links = get_links_set(reader)
+    i14y_doc[content_key] = f"{i14y_doc[content_key]} {" ".join(all_links) if len(all_links) > 0 else ""}"
+
+    return i14y_doc
path: search_gov_crawler/elasticsearch/convert_pdf_i14y.py
user: login: selfdanielj
body: I just thought it was more clear when you had the dict assignment at the end and all the field assignments happening there as well.
created_at: 2025-03-18T10:51:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -52,14 +102,21 @@ def convert_pdf(response_bytes: bytes, url: str, response_language: str = None):
         "tags": keywords,
         "updated_at": time_now_str,
         "updated": null_date(meta_values.get("CreationDate")),
-        f"title{valid_language}": title,
-        f"description{valid_language}": content.sanitize_text(description),
-        f"content{valid_language}": content.sanitize_text(main_content),
+        title_key: title,
+        description_key: content.sanitize_text(description),
+        content_key: content.sanitize_text(main_content),
         "basename": basename,
         "extension": extension or None,
         "url_path": get_url_path(url),
         "domain_name": get_domain_name(url),
     }
+
+    add_title_and_filename(content_key, title_key, i14y_doc)
+    add_title_and_filename(description_key, title_key, i14y_doc)
+    all_links = get_links_set(reader)
+    i14y_doc[content_key] = f"{i14y_doc[content_key]} {" ".join(all_links) if len(all_links) > 0 else ""}"
+
+    return i14y_doc
path: search_gov_crawler/elasticsearch/convert_pdf_i14y.py
user: login: igoristic
body: I see your point, but I think it becomes more of a personal preference/style at that point
created_at: 2025-03-18T15:13:51Z
author_association: CONTRIBUTOR

PR comments:
user: login: selfdanielj
created_at: 2025-03-17T20:43:31Z
body: Seems to work, I can see PDFs in the index!

I don't need any changes for my comments but I guess I'll just reiterate my concern about the complexity we are adding to the creation of the documents.  Fields go through multiple updates, multiple functions in multiple files, its getting hard to trace.  I'd just ask that as we make additions we look for ways to simplify things... maybe encapsulating all transformations for a given field into a single function? instead of having several ORs around the file? If I am getting annoyed at following some of these fields through all the options I can only imagine a greater difficulty for someone new coming to this.
author_association: CONTRIBUTOR
user: login: IsabelLaurenceau
created_at: 2025-03-18T11:42:32Z
body: > Seems to work, I can see PDFs in the index!
> 
> I don't need any changes for my comments but I guess I'll just reiterate my concern about the complexity we are adding to the creation of the documents. Fields go through multiple updates, multiple functions in multiple files, its getting hard to trace. I'd just ask that as we make additions we look for ways to simplify things... maybe encapsulating all transformations for a given field into a single function? instead of having several ORs around the file? If I am getting annoyed at following some of these fields through all the options I can only imagine a greater difficulty for someone new coming to this.

Agree. @igoristic can you create a new ticket to address some of Dan's concerns? I don't think we need to do that right now but I want them documented for future work so we don't lose them. 
author_association: CONTRIBUTOR