2_use_case_name: Automating Multilingual Census Data Processing: An AI and Transformer-Based Pipeline for Efficient Language Detection and Translation for Short-Text
3_agency: Department of Commerce
3_abr: DOC
4_bureau: CENSUS - U.S. Census Bureau
8_topic_area: Mission-Enabling (internal agency support)
10_commercial_ai: None of the above.
11_purpose_benefits: The purpose of this AI-driven pipeline is to tackle the challenge of processing non-English, short-text responses in large-scale surveys and censuses, especially brief entries for race and ethnicity. Traditional language detection and translation systems often struggle with these minimal text responses, impacting data accuracy and inclusivity. This AI solution is designed to automate the entire multilingual data processing workflow, from language detection to translation, named entity recognition, and validation, using AI and transformer-based models along with natural language processing techniques. By achieving high accuracy even with limited context, the system reduces the need for human translators, increases processing speed, and guarantees fair and accurate representation of diverse populations. This innovation not only supports the agency’s mission to collect inclusive, representative data but also benefits the public by contributing to more precise demographic insights, ultimately aiding in resource allocation and policy making.
12_outputs: The AI system outputs a series of automated decisions and validated translations for short-text responses, such as race and ethnicity write-ins, within large-scale survey data. Specifically, it provides accurate language detection, corrects potential spelling errors, generates contextually accurate translations, and validates these translations against standardized labels using semantic similarity analysis. The system then selects the most accurate translation for each input, guaranteeing precise categorization and extraction of demographic data. These outputs streamline survey data processing by reducing the need for human intervention, allowing for real-time or near real-time responses that meet high standards of accuracy and inclusivity
16_dev_stage: Acquisition and/or Development
17_impact_type: Neither
18_date_initiated: 6/3/2024
19_date_acq_dev_began: 7/8/2024
20_date_implemented: 9/30/2025
22_dev_method: Developed in-house.
24_hisp_support: No
33_agency_data: The agency used agency-owned decennial paradata to train, fine-tune, and evaluate the AI model’s performance. This paradata, collected from previous census responses, provided important interaction data that allowed the model to improve its language detection and translation accuracy, especially for short-text responses typical in race and ethnicity write-ins.
34_data_docs: Documentation is complete: Documentation exists regarding the maintenance, composition, quality, and intended use of the training and evaluation data, as well as any statistical bias across model features and protected groups.
35_demo_features: Race/Ethnicity
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
40_has_ato: No
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: None: This use case does not re-use any internally developed tooling or managed infrastructure from any other AI development efforts within the agency.
50_internal_review: Documentation has been developed: Complete documentation detailing model performance across a range of benchmarks, architecture, relevant features and information regarding the appropriate use of the model for predictive tasks has been created.