Bug: Spark SQL returns table column names as literal data values for Hive tables
Status: Open | Priority: Major | Resolution: 
Created: 03/Oct/23 23:01
Affects Version/s: 3.4.0

When using Spark SQL and Hive JDBC driver to access a Hive table the resulting row data is replaced with the literal column name in the resulting dataframe result.

When I run this:

jdbcDF = spark.read \
  .format("jdbc") \
  .options(driver="org.apache.hive.jdbc.HiveDriver",
           url="jdbc:hive2://10.20.174.171:10009",
           user="10009",
           password="123",
           query="select * from demo.hospitals limit 10"
           ) \
  .load()

 

I get:

+```+

+------------+---+----+-------+----+-----+---+-----------+---+---+-----+------------------+--------+---------+----+---+-----------+---------+---------------+------+ |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| +------------+---+----+-------+----+-----+---+-----------+---+---+-----+------------------+--------+---------+----+---+-----------+---------+---------------+------+ |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| +------------+---+----+-------+----+-----+---+-----------+---+---+-----+------------------+--------+---------+----+---+-----------+---------+---------------+------+

+```+

I should see:

```

+--------------------+--------------------+--------+-------------+-------------+-------+--------------------+--------------------+--------------------+------+-------+----------+---------+---------+ | person_pk| race_value|sex_code|poverty_value|veteran_value|ppr_pro| patient_pk| di_dk| pov_pk|vet_pk|veteran|total_paid|num_drugs|immunized| +--------------------+--------------------+--------+-------------+-------------+-------+--------------------+--------------------+--------------------+------+-------+----------+---------+---------+ |001252a7-a1e7-428...|01 - American Ind...| F| 37.0| null| 2|65007233-424e-4c2...|9d66f5b7-ab10-47f...|1f3d76c8-d039-483...| |unknown| null| null| true| |002673d4-579a-4d1...|01 - American Ind...| M| 64.0| null| 2|a3c89a7f-d57d-4be...|2f6ffa09-e5b3-419...|7dbfc730-64bc-4a9...| |unknown| null| null| true| |00267822-8192-44f...|01 - American Ind...| F| 0.0| null| 2|cd318b72-35d4-422...|44646492-60ef-44e...|d5f462ef-cd4c-497...| |unknown| null| null| true| |0028fece-59ec-4db...|01 - American Ind...| F| 0.0| null| 2|ee9e09aa-67bc-47e...|3be068de-7fe3-44d...|63a04010-c381-4aa...| |unknown| null| null| true| |003470e7-b548-444...|06 - American Ind...| M| 171.0| null| 2|7ed5b0f9-02b3-459...|1b778c9f-71ab-45a...|84ecc23a-6c39-44d...| |unknown| null| null| false| |0044a493-e226-409...|01 - American Ind...| F| 0.0| null| 2|c821f5b2-d0af-428...|26144dac-81f0-44e...|f7355eeb-89a3-4f0...| |unknown| null| null| true| |004d44d0-fdf7-403...|01 - American Ind...| F| 37.0| null| 2|cb6c8e5c-71ab-409...|88eaf3c4-5f00-4e9...|78679644-f4e7-450...| |unknown| null| null| true| |0059c1bf-5263-42a...|03 - Black or Afr...| M| 0.0| null| 2|da9247d1-96fb-44d...|6831544a-faf9-426...|3534f3a8-a367-41e...| |unknown| null| null| true| |007b82b6-ae2e-49e...|01 - American Ind...| M| 43.0| null| 2|3e6fcc8c-c484-465...|90e2a03f-f0a4-48f...|5c9c71e1-901b-481...| |unknown| null| null| true| |00917cf5-d879-43f...|01 - American Ind...| F| 0.0| null| 2|b4fa8b2f-7452-4f5...|779eaa9e-7961-4f0...|2d19ad6b-de35-4bc...| |unknown| null| null| true| +--------------------+--------------------+--------+-------------+-------------+-------+--------------------+--------------------+--------------------+------+-------+----------+---------+---------+

```

(This is de-identified data)