Bug: count distinct asterisk 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/May/21 17:19
Affects Version/s: 3.0.0

Hi everyone,

I hope you're well!

 

Today I came across a very interesting case when the result of the execution of the algorithm for counting unique rows differs depending on the form (count(distinct *) vs count( * ) from derived table) of the Spark SQL queries.

I still can't figure out on my own if this is a bug or a feature and I would like to share what I found.

 

I run Spark SQL queries through the Thrift (and not only) connecting to the Spark cluster. I use the DBeaver app to execute Spark SQL queries.

 

So, I have two identical Spark SQL queries from an algorithmic point of view that return different results.

 

The first query:
{code:sql}
select count(distinct *) unique_amt from storage_datamart.olympiads
; -- Rows: 13437678
{code}
 

The second query:
{code:sql}
select count(*) from (select distinct * from storage_datamart.olympiads)
; -- Rows: 36901430
{code}
 

The result of the two queries is different. (But it must be the same, right!?)
{code:sql}
select 'The first query' description, count(distinct *) unique_amt from storage_datamart.olympiads
 union all
select 'The second query', count(*) from (select distinct * from storage_datamart.olympiads)
;
{code}
 

The result of the above query is the following:
{code:java}
The first query    13437678
The second query   36901430
{code}
 
 I can easily calculate the unique number of rows in the table:
{code:sql}
select count(*) from (
  select student_id, olympiad_id, tour, grade
    from storage_datamart.olympiads
   group by student_id, olympiad_id, tour, grade
  having count(*) = 1
)
; -- Rows: 36901365
{code}
 

The table DDL is the following:
{code:sql}
CREATE TABLE `storage_datamart`.`olympiads` (
  `ptn_date` DATE,
  `student_id` BIGINT,
  `olympiad_id` STRING,
  `grade` BIGINT,
  `grade_type` STRING,
  `tour` STRING,
  `created_at` TIMESTAMP,
  `created_at_local` TIMESTAMP,
  `olympiad_num` BIGINT,
  `olympiad_name` STRING,
  `subject` STRING,
  `started_at` TIMESTAMP,
  `ended_at` TIMESTAMP,
  `region_id` BIGINT,
  `region_name` STRING,
  `municipality_name` STRING,
  `school_id` BIGINT,
  `school_name` STRING,
  `school_status` BOOLEAN,
  `oly_n_common` INT,
  `num_day` INT,
  `award_type` STRING,
  `new_student_legacy` INT,
  `segment` STRING,
  `total_start` TIMESTAMP,
  `total_end` TIMESTAMP,
  `year_learn` STRING,
  `parent_id` BIGINT,
  `teacher_id` BIGINT,
  `parallel` BIGINT,
  `olympiad_type` STRING)
USING parquet
LOCATION 's3a://uchiru-bi-dwh/storage/datamart/olympiads.parquet'
;
{code}
 

Could you please tell me why in the first Spark SQL query counting the unique number of rows using the construction `count(distinct *)` does not count correctly and why the result of the two Spark SQL queries is different??

Thanks in advance.

 

p.s. I could not find a description of such behaviour of the function `count(distinct *)` in the [official Spark documentation|https://spark.apache.org/docs/latest/sql-ref-functions-builtin.html#aggregate-functions]:
{quote}count(DISTINCT expr[, expr...]) -> Returns the number of rows for which the supplied expression(s) are unique and non-null.
{quote}