There is a score column in a dataset.  Your goal is to come up with a formula that estimates the score value from the values of the other columns.   We have generated the dataset so that there is a formula that, for each borrower, accurately calculates the score value from the values of the other columns. Finding an accurate formula can be tricky, so you need to find a formula that estimates the score as accurately as possible.   Your formula can use the values of any of the 20 columns that describe the borrower, from name to util_flg. The output of the formula should return a single number for each borrower: the score estimate. Example formula in pandas: result = (df[education_level_code'].map({ 'SPECIALTY': 0.2, 'BACHELOR': 0.3, 'MASTER': 0.5}) + 1 - df['approve_flg']) / 2 Here the education level is encoded with a real number, then one is added to it and the value of the approval flag is subtracted. The final result is divided by two.   What you can and can't use In your solution, you can use:  - any elementary functions and rounding functions - addition, multiplication, subtraction and division operations applied to columns element by element - substitutions: replacing character expressions with numbers, e.g. “MASTER” → 0.3 - property counting functions: number of characters in a string, day of the week for a date, ...  - any pandas and numpy functions You need to respect some restrictions:  - you can use machine learning algorithms to explore the dataset, but your resulting formula must depend only on the values of the original features and use only the functions described above - your formula must work only with the values of the twenty columns describing the borrower: you cannot refer to row number, other borrowers,...   Important: In the formula, all columns must be called by real names (name, create_dttm, etc.) Solution Evaluation The solution will be evaluated on two criteria:
- Formula accuracy. The MAE (Mean Absolute Error) metric will be used. All solutions will be ordered by increasing error - the higher the participant's ranking (i.e., the lower the error), the more points he/she will receive.
- Report Quality. We will evaluate the completeness and clarity of the report, as well as the originality of the methods applied. In addition, the compactness and simplicity of the formula is evaluated - try to find a simple and beautiful solution.

Solution submission
To submit your solution you need to do three steps.
Step 1: Calculate the error of your formula and round it to the third decimal place according to the rules of mathematics. Enter the resulting value in the first line of the text box on the solution form.
Step 2. Write your final formula in a txt file and attach it to the solution. If you are solving the problem in pandas, copy the final formula and paste it into the file. If you're solving the problem in Excel or Google Sheets, describe the formula in pseudocode.
Step 3: Write a report with a precise description of the final formula and the solution: the logic of matching column names, the steps of formula search, the list of considered solutions. You may cite code fragments in the report. If you used any sources of information, list them. The report should be organized in one pdf file without references to additional parts. File size: no more than 5 A4 pages.
Important notes
* We will manually check the accuracy of your formula. If the value we get diverges from the value you specify in step 1, you will receive a penalty. 
* We will check the accuracy of your formula not only on the data you are working with, but also on closed test data. The test data is generated using the same principle as the main dataset, so your formula should show similar results. If the accuracy of your formula deteriorates dramatically, you will receive a penalty. 
* We will apply your formula to a dataset in which all columns are named correctly. If you get the column names wrong, your formula gets a low score. 

 To solve this problem, I wrote this code:
import pandas as pd
import numpy as np

!pip install catboost

!pip install optuna

column_names = [
    "approve_dttm", "utm_source", "specialty", "util_dttm", "semester_cost_amt", "pid",
    "score", "util_flg", "approve_flg", "semestr_cnt", "short_nm", "initial_approved_amt",
    "reject_reason", "gender_cd", "initial_term", "name", "marketing_flag",
    "education_level_code", "age", "subside_rate", "create_dttm"
]

data = pd.read_csv('/content/sample_data/credit-c61fbdb3-e0af-4b7b-9301-6ba7a994eeca.csv', header=None, names=column_names)
data = data.drop(index=0).reset_index(drop=True)

data['age'] = pd.to_numeric(data['age'], errors='coerce').fillna(0).astype(int)
data['subside_rate'] = pd.to_numeric(data['subside_rate'], errors='coerce').fillna(0).astype(int)
data['approve_flg'] = pd.to_numeric(data['approve_flg'], errors='coerce').fillna(0).astype(int)


# Assuming `data` is the dataset
# Map categorical education level to numeric values
data['education_level_num'] = data['education_level_code'].map({'SPECIALTY': 0.2, 'BACHELOR': 0.3, 'MASTER': 0.5}).fillna(0)

# Create time-based features
data['days_to_approve'] = (pd.to_datetime(data['approve_dttm']) - pd.to_datetime(data['create_dttm'])).dt.days
data['days_to_util'] = (pd.to_datetime(data['util_dttm']) - pd.to_datetime(data['approve_dttm'])).dt.days

# Create interaction terms and polynomial features
data['age_squared'] = data['age'] ** 2
data['education_age_interaction'] = data['education_level_num'] * data['age']
data['subside_rate_squared'] = data['subside_rate'] ** 2

from xgboost import XGBRegressor
from sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV
from sklearn.metrics import mean_absolute_error


X = data[features]
y = data['score']

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initial model
xgb = XGBRegressor(objective='reg:squarederror', random_state=42)

# Parameter grid for GridSearchCV
# Fine-tuning example using a narrow range for XGBoost
param_grid = {
    'learning_rate': [0.03, 0.04, 0.05],
    'max_depth': [4, 5, 6],
    'n_estimators': [250, 300, 350],
    'subsample': [0.85, 0.9, 0.95],
    'colsample_bytree': [0.75, 0.8, 0.85]
}

# Repeat GridSearchCV or RandomizedSearchCV with these parameters to further optimize

# Hyperparameter tuning
grid_search = GridSearchCV(estimator=xgb, param_grid=param_grid, cv=5, scoring='neg_mean_absolute_error', verbose=1, n_jobs=-1)
grid_search.fit(X_train, y_train)

# Best model
best_xgb = grid_search.best_estimator_

# Predictions and MAE
y_pred_train = best_xgb.predict(X_train)
y_pred_test = best_xgb.predict(X_test)
mae_train = mean_absolute_error(y_train, y_pred_train)
mae_test = mean_absolute_error(y_test, y_pred_test)

print("Best Model MAE on Training Set:", mae_train)
print("Best Model MAE on Test Set:", mae_test)
print("Best Model Parameters:", grid_search.best_params_)  

However, it only allows me to get such an MAE:
 Best Model MAE on Training Set: 0.0536856410432385
Best Model MAE on Test Set: 0.055032521931449876
Best Model Parameters: {'colsample_bytree': 0.8, 'learning_rate': 0.05, 'max_depth': 3, 'n_estimators': 100, 'subsample': 0.8}  
While my goal to lower MAE is less than 0.02