Problem: CGSS2013 Data Processing and Variable Creation

The problem involves processing CGSS2013.dta by cleaning demographic, educational, and organizational variables, generating indicators for party membership, region, winsorized income, and manager status, then creating occupation categories/dummies and removing incomplete observations to yield a cleaned dataset ready for analysis.


Input/Output: The problem takes the CGSS2013.dta dataset as input, which contains raw variables including a3a (birth year), isurban (urban/rural indicator), a58 (residence type), a2 (sex), a7a (education level), a10 (party membership), a59j/a59k (organization type), s41 (region code), a18 (rural residence), a8a (2013 income), and a59f (manager status). It processes the data by loading the file, cleaning age (derived from a3a, filtered to 18–60 years), sex (recoded to a female dummy), education (converted to years of schooling), filtering valid observations for party membership, organization type (recoded into categories and dummy variables), region (categorized into three_municipal_cities, central, west, and east), rural status, winsorized income (1% and 99% percentiles) and its natural log, and manager status. It then generates occupation categories (based on manager status and organization type) and corresponding dummy variables (manager_state, manager_nonstate, self_employed, worker), and removes observations with missing values in predefined variables (age, age2, female, rural, region dummies, edu_years, party_member, occupation dummies, organization dummies, lnincome). The problem outputs a cleaned Stata dataset with all processed variables, aligned with standard CGSS2013 data processing logic.

Required dependencies:

Sub-steps:
step_number: 1
step_description_prompt: Please write Stata code to create a data preprocessing workflow named cgss2013_data_preprocessing (using standard Stata syntax without `program define` structure)

1. Function Definition
This workflow is designed to perform comprehensive data cleaning, variable recoding, new variable generation, and sample filtering on the CGSS2013 dataset, preparing it for subsequent statistical analysis. All operations follow strict rules for handling missing values, invalid codes, and variable transformation.

2. Algorithm Flow
Step 1: Configure Stata environment: Disable more output and set `matsize` to 1000.
Step 2: Load the dataset from the path "../test_data/CGSS2013.dta".
Step 3: Process birth year and age variables:
   - Set values of `a3a` (birth year) to missing if they are less than 1900 or greater than 2013.
   - Generate `age` as 2013 minus `a3a`.
   - Keep only samples where `age` is between 18 and 60 (inclusive).
   - Generate `age2` as the square of `age`.
Step 4: Filter samples:
   - Keep only samples where `isurban` equals 2.
   - Keep only samples where `a58` is in the list [1, 2, 3].
Step 5: Recode gender variables:
   - Rename `a2` to `sex`.
   - Recode `sex` values: set 1 to 0, set 2 to 1.
   - Generate `female` variable equal to the recoded `sex`.
Step 6: Generate education years variable (`edu_years`):
   - Assign 6 if `a7a` is in [1,2,3]
   - Assign 9 if `a7a` equals 4
   - Assign 12 if `a7a` is in [5,6,7,8]
   - Assign 16 if `a7a` is in [9,10]
   - Assign 19 if `a7a` is in [11,12,13]
Step 7: Process party membership variable (`party_member`):
   - Set to 1 if `a10` equals 1.
   - Set to 0 if `a10` is in [2,3,4].
   - Set to missing if `a10` is less than 0.
   - Drop samples where `party_member` is missing.
Step 8: Process organization-related variables:
   - Drop samples where `a59j` is negative, missing, or `a59k` is negative.
   - Generate `organization` variable by recoding `a59j` and `a59k`:
     - `organization` = 1 if `a59j` is 1 or 6
     - `organization` = 2 if `a59j` equals 3 and `a59k` equals 1
     - `organization` = 3 if `a59j` equals 2 and `a59k` equals 1
     - `organization` = 4 if (`a59j` equals 2 and `a59k` equals 2) or (`a59j` equals 3 and `a59k` equals 2)
     - `organization` = 5 if (`a59j` equals 2 and `a59k` is between 3 and 5 inclusive) or (`a59j` equals 3 and `a59k` is between 3 and 5 inclusive) or `a59j` equals 5
     - `organization` = 6 if `a59j` equals 4 or 7
   - Generate binary variables `Goverment_agency`, `Pubic_organization`, `Collective_firm`, `Private_firm`, `No_organization_affiliation` where each variable equals 1 when `organization` matches the corresponding category (1, 2, 4, 5, 6 respectively), 0 otherwise.
Step 9: Generate regional binary variables:
   - `three_municipal_cities` = 1 if `s41` is in [1,4,7]
   - `central` = 1 if `s41` is in [9,11,16,18,21,22]
   - `west` = 1 if `s41` is in [2,3,6,8,13,23,26,28,29,30]
   - `east` = 1 if `s41` is in [5,10,12,15,17,19,24,27,31]
Step 10: Generate `rural` variable: set to 1 if `a18` equals 1, 0 otherwise.
Step 11: Process income variables:
   - Rename `a8a` to `income2013`.
   - Perform winsorization at the 1st and 99th percentiles of `income2013` to create `income_winsor`: replace values below the 1st percentile with the 1st percentile value, replace values above the 99th percentile with the 99th percentile value.
   - Drop samples where `income_winsor` is 0 or greater than 800000.
   - Generate `lnincome` as the natural logarithm of `income_winsor`.
Step 12: Process manager variable:
   - Drop samples where `a59f` equals -3.
   - Generate `manager` variable by recoding `a59f`:
     - `manager` = 1 if `a59f` is 1 or 2
     - `manager` = 2 if `a59f` equals 3
     - `manager` = 3 if `a59f` equals 4

3. Mathematical Principles
- Age calculation: age = 2013 - a3a, a simple subtraction to derive age from birth year.
- Age squared: age2 = age^2, a quadratic transformation to capture non-linear age effects.
- Winsorization: replaces extreme values (below 1st percentile and above 99th percentile) with percentile values to mitigate the impact of outliers on subsequent analysis.
- Natural logarithm transformation: lnincome = ln(income_winsor), applied to income to normalize its distribution and reduce skewness.
- Binary variable generation: uses logical condition checks to assign 1 (true) or 0 (false) based on specified criteria for categorical variables.

4. Technical Requirements
- Implement all operations using standard Stata commands (no `program define` allowed).
- Ensure all variable names match exactly as specified.
- Strictly follow all recoding rules for each variable.
- Use appropriate Stata commands such as `replace`, `gen`, `keep`, `drop`, `inlist`, `_pctile` as needed.
- Handle missing values and invalid negative codes correctly per variable rules.

5. Notes
- Do not add any additional operations or variables not specified in the workflow.
- Ensure all sample filtering conditions are strictly applied to retain only eligible samples.
- Verify that all missing values are handled appropriately (set to missing or drop samples as specified).
- Ensure that all recoding steps are implemented precisely to avoid errors in variable creation.
- Do not provide the complete Stata code; instead, define all steps with sufficient detail to fully reproduce the original preprocessing workflow.
test_cases: count
local final_obs = r(N)

sum age
local mean_age = r(mean)
local sd_age = r(sd)

sum lnincome, detail
local median_lnincome = r(p50)
local mean_lnincome = r(mean)

sum female
local prop_female = r(mean)

sum party_member
local prop_party = r(mean)

local verification_value = `final_obs' + `mean_age'*10 + `median_lnincome'*100 + `prop_female'*1000 + `prop_party'*10000

local tolerance = 0.001
scalar correct = 6487.439
local diff = abs(`verification_value' - correct)
if `diff' > `tolerance'{
 display as error "Wrong"
 exit 1
}
else{
 display "Completely right!"
}

step_number: 2
step_description_prompt: Please write Stata code to create a function named process_occupation_and_clean_data
2. Function Definition: This function is designed to derive occupation-related categorical and binary indicator variables from existing 'manager' and 'organization' variables, followed by listwise deletion of observations with missing values for a predefined set of variables.
3. Algorithm Flow:
1. Initialize a new variable named 'occupation' with missing values.
2. Assign values to 'occupation' using conditional statements:
   a. Set occupation = 1 when manager == 1 and organization is in the set {1, 2, 3}
   b. Set occupation = 2 when manager == 1 and organization is in the set {4, 5, 6}
   c. Set occupation = 3 when manager == 3
   d. Set occupation = 4 when manager == 2
3. Generate four binary indicator variables using logical checks on 'occupation':
   a. 'manager_state' which equals 1 if occupation equals 1, else 0
   b. 'manager_nonstate' which equals 1 if occupation equals 2, else 0
   c. 'self_employed' which equals 1 if occupation equals 3, else 0
   d. 'worker' which equals 1 if occupation equals 4, else 0
4. Iterate over a specified list of variables using a loop, and drop observations where each variable has a missing value.
4. Mathematical Principles:
- Categorical variable assignment follows discrete conditional mapping based on input variable values.
- Binary indicator variables are constructed using the indicator function I(condition), where I returns 1 if the condition is satisfied and 0 otherwise.
- Listwise deletion applies the principle of excluding incomplete records by removing observations where any of the target variables have missing values (denoted as . in Stata).
5. Technical Requirements:
- Use standard Stata command syntax; do not use the 'program define' structure.
- Employ the 'gen' command to initialize new variables and create binary indicators.
- Use the 'replace' command to assign values to the 'occupation' variable based on conditional logic.
- Utilize the 'inlist()' function to check if 'organization' is within specified value sets.
- Implement a 'foreach' loop to iterate over the predefined variable list for missing value handling.
6. Notes:
- Do not add any functionality that is not specified in the algorithm flow.
- Ensure all conditional rules for 'occupation' are strictly followed without modification.
- Include all variables in the following list in the foreach loop: age, age2, female, rural, three_municipal_cities, central, west, east, edu_years, party_member, manager_state, manager_nonstate, self_employed, Goverment_agency, Pubic_organization, Collective_firm, Private_firm, No_organization_affiliation, lnincome
- Maintain the exact variable names as specified in all steps.
- Ensure that after the missing value deletion step, none of the specified variables contain missing values.
test_cases: clear
set obs 500

gen manager = 1 in 1/100
replace manager = 2 in 101/300
replace manager = 3 in 301/500
gen organization = 1 in 1/150
replace organization = 5 in 151/350
replace organization = 6 in 351/500

gen occupation = .
replace occupation = 1 if manager == 1 & inlist(organization, 1, 2, 3)
replace occupation = 2 if manager == 1 & inlist(organization, 4, 5, 6)
replace occupation = 3 if manager == 3
replace occupation = 4 if manager == 2

gen manager_state = (occupation == 1)
gen manager_nonstate = (occupation == 2)
gen self_employed = (occupation == 3)
gen worker = (occupation == 4)

count
local N = r(N)
sum occupation
local occ_mean = r(mean)
sum manager_state
local m_state = r(mean)

local val = `N' + `occ_mean'*112 + `m_state'*1260

local tolerance = 0.001
scalar correct = 1088.000
local diff = abs(`val' - correct)
if `diff' > `tolerance'{
 display as error "Wrong"
 exit 1
}
else{
 display "Completely right!"
}

General solution:

General tests: