Databricks ( Spark ) にてデータフレームのカラム名を一括で変更する方法

## 概要

Databricks ( Spark ) にてデータフレームのカラム名を一括で変更する方法を共有します。

`withColumnRenamed` 関数によりカラム名を変更できますが、多数のカラムが場合などには次のような関数を利用することがおすすめです。

```python
def rename_df_cols(
    df,
    cols_names,
):
    counter = 0
    for new_col in cols_names:
        counter_str = str(counter)
        old_col_name = df.columns[counter]
        df = df.withColumnRenamed(old_col_name, new_col)
        counter += 1
    return df
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/24031/4607de96-d5c6-fc14-5f30-a6b74c117c71.png)


CSV をソースとしたデータフレームを `header` を `False` とした作成した場合に`_#` というカラム名になるため、上記の関数が役立ちます。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/24031/177a02d9-1422-55f1-3b5e-f4ce7ebada03.png)


## 実行例
### 事前準備

```python
import inspect
from pyspark.sql import DataFrame, SparkSession
from pyspark.dbutils import DBUtils
 
def put_files_to_storage(
    path,
    content,
    is_overwrite = True,
):
    # 最初の行と最後の行を削除
    content = inspect.cleandoc(content)
   
    spark = SparkSession.getActiveSession()
    dbutils = DBUtils(spark)
    return dbutils.fs.put(path, content, is_overwrite)

# 検証用ファイルの配置
path = 'dbfs:/test/test.csv'
 
csv_data_rows = """
1,"user_aaa","user_aaa@test.com",53.5,180
2,"user_bbb","user_bbb@test.com",53.5,180
3,"user_ccc","user_ccc@test.com",53.5,180
"""
 
_ = put_files_to_storage(path, csv_data_rows)
 
# ファイルを確認
file_content = dbutils.fs.head(path)
print(file_content)
Wrote 125 bytes.
1,"user_aaa","user_aaa@test.com",53.5,180
2,"user_bbb","user_bbb@test.com",53.5,180
3,"user_ccc","user_ccc@test.com",53.5,180
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/24031/491317ab-6a32-d32a-8e20-f2f9873f6c66.png)

```python
# データフレームを作成
df = (
    spark
    .read
    .format('csv')
    .option("inferSchema", "False")
    .option('header', 'False')
    .load(path)
)
df.display()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/24031/32c8bb16-cf33-0779-e9c5-a12a02db6995.png)


### 1. カラムのリスト型変数により置換する方法

```python
def rename_df_cols(
    df,
    cols_names,
):
    counter = 0
    for new_col in cols_names:
        counter_str = str(counter)
        old_col_name = df.columns[counter]
        df = df.withColumnRenamed(old_col_name, new_col)
        counter += 1
    return df

```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/24031/3e4b1b5c-19d1-a506-2192-e0b0aaad2204.png)


```python
col_names = ['id','name','mail','weight','height',]

df_2 = rename_df_cols(df, col_names)
```

```python
df_2.display()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/24031/306e0f58-476b-9d45-a456-bb40ae4618da.png)


### 2. カラムの辞書型変数により置換する方法

```python
def rename_df_cols_with_col_map(
    df,
    renamed_cols_names,
):
    for existing_col,new_col in renamed_cols_names.items():
        df = df.withColumnRenamed(existing_col, new_col)
    return df
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/24031/b68450f7-3534-7a80-5101-0670581cc57b.png)

```python
# ソースファイルにヘッダーがないため、カラム名を変更
renamed_cols_names = {
    '_c0':'id',
    '_c1':'name',
    '_c2':'mail',
    '_c3':'weight',
    '_c4':'height',
}
```

```python
df_3 = rename_df_cols_with_col_map(df, renamed_cols_names)
 
df_3.display()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/24031/445e8e1b-d1a1-c017-f44b-56021ebdf69d.png)

### 事後処理

```python
dbutils.fs.rm(path, True)
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/24031/4883431d-9c13-fa33-9d81-0aee61f05511.png)