av9ash/GitBugs huggingface dataset:
Bug: Add higher order functions API to PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jan/20 13:16
Affects Version/s: 3.0.0

As of 3.0.0 higher order functions are available in SQL and Scala, but not in PySpark, forcing Python users to invoke these through {{expr}}, {{selectExpr}} or {{sql}}.

This is error prone and not well documented. Spark should provide {{pyspark.sql}} wrappers that accept plain Python functions (of course within limits of {{(*Column) -> Column}}) as arguments.


{code:python}
df.select(transform("values", lambda c: trim(upper(c)))

def  increment_values(k: Column, v: Column) -> Column:
    return v + 1

df.select(transform_values("data"), increment_values)
{code}
Bug: Add higher order functions API to SparkR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jan/20 13:21
Affects Version/s: 3.0.0

As of 3.0.0 higher order functions are available in SQL and Scala, but not in SparkR forcing R users to invoke these through {{expr}}, {{selectExpr}} or {{sql}}.

It would be great if Spark provided high level wrappers that accept plain R functions operating on SQL expressions.
Bug: upgrade testthat on jenkins workers to 2.0.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/20 19:17
Affects Version/s: 3.0.0

see:  https://issues.apache.org/jira/browse/SPARK-23435

i will investigate upgrading testthat on my staging worker, and if that goes smoothly we can upgrade it on all jenkins workers.
Bug: Pyspark test "test_memory_limit" fails consistently
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/20 05:11
Affects Version/s: 3.0.0

I'm seeing consistent pyspark test failures on multiple PRs ([#26955|https://github.com/apache/spark/pull/26955], [#26201|https://github.com/apache/spark/pull/26201], [#27064|https://github.com/apache/spark/pull/27064]), and they all failed from "test_memory_limit".

[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116422/testReport]

[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116438/testReport]

[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116429/testReport]

[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116366/testReport]
Bug: Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF
Status: In Progress | Priority: Major | Resolution: 
Created: 30/Jan/20 23:04
Affects Version/s: 2.3.2

 
{code:java}
scala> spark.sql("select unix_timestamp('20201', 'yyyyww')").show();
+-----------------------------+
|unix_timestamp(20201, yyyyww)|
+-----------------------------+
|                         null|
+-----------------------------+
 
scala> spark.sql("select unix_timestamp('20202', 'yyyyww')").show();
-----------------------------+
|unix_timestamp(20202, yyyyww)|
+-----------------------------+
|                   1578182400|
+-----------------------------+
 
{code}
 

 

This seems to happen for leap year only, I dig deeper into it and it seems that  Spark is using the java.text.SimpleDateFormat and try to parse the expression here

[org.apache.spark.sql.catalyst.expressions.UnixTime#eval|https://github.com/hortonworks/spark2/blob/49ec35bbb40ec6220282d932c9411773228725be/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L652]
{code:java}
formatter.parse(
 t.asInstanceOf[UTF8String].toString).getTime / 1000L{code}
 but fail and SimpleDateFormat unable to parse the date throw Unparseable Exception but Spark handle it silently and returns NULL.

 

*Spark-3.0:* I did some tests where spark no longer using the legacy java.text.SimpleDateFormat but java date/time API, it seems  date/time API expect a valid date with valid format

 org.apache.spark.sql.catalyst.util.Iso8601TimestampFormatter#parse
Bug: When reading from a file with pre-defined schema and encountering a single value that is not the same type as that of its column , Spark nullifies the entire row
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 30/Jan/20 22:59
Affects Version/s: 2.3.0

When reading from a file with pre-defined schema and encountering a single value that is not the same type as that of its column , Spark nullifies the entire row instead of setting the value at that cell to be null.

 
{code:java}
case class TestModel(
  num: Double, test: String, mac: String, value: Double

)
val schema = ScalaReflection.schemaFor[TestModel].dataType.asInstanceOf[StructType]
//here's the content of the file test.data
//    1~test~mac1~2
//    1.0~testdatarow2~mac2~non-numeric
//    2~test1~mac1~3
val ds = spark
  .read
  .schema(schema)
  .option("delimiter", "~")
  .csv("/test-data/test.data")
ds.show();
//the content of data frame. second row is all null. 
//      +----+-----+----+-----+
//      | num| test| mac|value|
//      +----+-----+----+-----+
//      | 1.0| test|mac1|  2.0|
//      |null| null|null| null|
//      | 2.0|test1|mac1|  3.0|
//      +----+-----+----+-----+

//should be

// +----+--------------+----+-----+ 
// | num| test         | mac|value| 
// +----+--------------+----+-----+ 
// | 1.0| test         |mac1| 2.0 | 
// |1.0 |testdatarow2  |mac2| null| 
// | 2.0|test1         |mac1| 3.0 | 
// +----+--------------+----+-----+{code}
Bug: Support filters pushdown in JSON datasource
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/20 11:43
Affects Version/s: 3.1.0

* Implement the `SupportsPushDownFilters` interface in `JsonScanBuilder`
 * Apply filters in JacksonParser
 * Change API JacksonParser - return Option[InternalRow] from `convertObject()` for root JSON fields.
 * Update JSONBenchmark
Bug: Add DSL functions invoking percentile_approx
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jan/20 12:48
Affects Version/s: 3.1.0

Since 2.1  (SPARK-16283)  Spark provides {{percentile_approx}}  SQL function, however it is not exposed through DSL, with exception to `DataFrameStatFunctions.approxQuantile` (which is eager and doesn't support grouping).

It would be useful to have it exposed through {{o.a.s.sql.functions}} / {{pyspark.sql.functions}}.
Bug: SparkSession emptyDataFrame should not create an RDD
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/20 21:57
Affects Version/s: 2.4.4

SparkSession.emptyDataFrame uses an empty RDD in the background. This is a bit of a pity because the optimizer can't recognize this as being empty and it won't apply things like {{PropagateEmptyRelation}}.
Bug: transform_keys function throws exception as "Cannot use null as map key", but there isn't any null key in the map 
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 27/Jan/20 06:19
Affects Version/s: 3.0.0

Have started experimenting Spark 3.0 new SQL functions and along the way found an issue with the *transform_keys* function. It is raising "Cannot use null as map key" exception but the Map actually doesn't hold any Null values.

Find my spark code below to reproduce the error.
{code:java}
val df = Seq(Map("EID_1"->10000,"EID_2"->25000)).toDF("employees")
df.withColumn("employees",transform_keys($"employees",(k,v)=>lit(k.+("XYX"))))
   .show
{code}
Exception in thread "main" java.lang.RuntimeException: *Cannot use null as map key*.
Bug: When creating a custom datasource File NotFoundExpection happens
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 27/Jan/20 07:19
Affects Version/s: 2.3.2

Hello, I'm creating a datasource based on FileFormat and DataSourceRegister. 

when I pass a path or a file that has a white space it seems to fail wit the error: 
{code:java}
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 213, localhost, executor driver): java.io.FileNotFoundException: File file:somePath/0019_leftImg8%20bit.png does not exist It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved. at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125) at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
{code}
I'm happy to fix this if someone tells me where I need to look.  

I think it is on org.apache.spark.rdd.InputFileBlockHolder : 
{code:java}
inputBlock.set(new FileBlock(UTF8String.fromString(filePath), startOffset, length))
{code}
Bug: The parquet file written by spark often incurs corrupted footer and hence not readable 
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 27/Jan/20 12:43
Affects Version/s: 1.6.1

This issue is similar to an archived one,

[https://mail-archives.apache.org/mod_mbox/spark-issues/201501.mbox/%3CJIRA.12767358.1421214067000.78480.1421214094403@Atlassian.JIRA%3E]

The parquet file written by spark often incurs corrupted footer and hence not readable by spark.

The issue is more consistent when the granularity of a field increases. i.e. when redundancy of values in dataset is reduced(= more number of unique values).

Coalesce also doesn't help here. It automatically generated a certain number of parquet files, each with a definite size as controlled by spark internals. But, few of them written corrupted footer. But writing job ends with success status. 

Here are few examples,

There are the files(267.2 M each) which the 1.6.x version spark has generated. But few of them are found with corrupted footer and hence not readable. This scenario happens more frequently when the file(input) size exceeds a certain limit and also the level of redundancy of the data matters. With the same file size, Lesser the level of redundancy, more is the probability of getting the footer corrupted.

Hence in iterations of the job when those are required to read for processing, ends up with
{{{color:#FF0000}*Can not read value 0 in block _n_ in file xxxx*{color}}}
Bug: collect() support Unicode charactes tests fails on Windows
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/20 00:46
Affects Version/s: 3.0.0

As-is [test_that("collect() support Unicode characters"|https://github.com/apache/spark/blob/d5b92b24c41b047c64a4d89cc4061ebf534f0995/R/pkg/tests/fulltests/test_sparkSQL.R#L850-L869] case seems to be system dependent, and doesn't work properly on Windows with CP1252 English locale:

 
{code:r}
library(SparkR)
SparkR::sparkR.session()
Sys.info()
#           sysname           release           version 
#         "Windows"      "Server x64"     "build 17763" 
#          nodename           machine             login 
# "WIN-5BLT6Q610KH"          "x86-64"   "Administrator" 
#              user    effective_user 
#   "Administrator"   "Administrator" 

Sys.getlocale()

# [1] "LC_COLLATE=English_United States.1252;LC_CTYPE=English_United States.1252;LC_MONETARY=English_United States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252"

lines <- c("{\"name\":\"안녕하세요\"}",
           "{\"name\":\"您好\", \"age\":30}",
           "{\"name\":\"こんにちは\", \"age\":19}",
           "{\"name\":\"Xin chào\"}")

system(paste0("cat ", jsonPath))
# {"name":"<U+C548><U+B155><U+D558><U+C138><U+C694>"}
# {"name":"<U+60A8><U+597D>", "age":30}
# {"name":"<U+3053><U+3093><U+306B><U+3061><U+306F>", "age":19}
# {"name":"Xin chào"}
# [1] 0


jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp")
writeLines(lines, jsonPath)

df <- read.df(jsonPath, "json")


printSchema(df)
# root
#  |-- _corrupt_record: string (nullable = true)
#  |-- age: long (nullable = true)
#  |-- name: string (nullable = true)

head(df)
#              _corrupt_record age                                     name
# 1                       <NA>  NA <U+C548><U+B155><U+D558><U+C138><U+C694>
# 2                       <NA>  30                         <U+60A8><U+597D>
# 3                       <NA>  19 <U+3053><U+3093><U+306B><U+3061><U+306F>
# 4 {"name":"Xin ch<U+FFFD>o"}  NA                                     <NA>

{code}

Problem becomes visible on AppVoyer when testthat is updated to 2.x, but somehow silenced when testthat 1.x is used.
Bug: Project Matrix: Linear Models revisit and refactor
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 25/Jan/20 13:12
Affects Version/s: 3.1.0

We had been refactoring linear models for a long time, and there still are some works in the future. After some discussions among [~huaxingao] [~srowen] [~weichenxu123] [~mengxr] [~podongfeng] , we decide to gather related works under a sub-project Matrix, it includes:
 # *Blockification (vectorization of vectors)*
 ** vectors are stacked into matrices, so that high-level BLAS can be used for better performance. (about ~3x faster on sparse datasets, up to ~18x faster on dense datasets, see SPARK-31783 for details).
 ** Since 3.1.1, LoR/SVC/LiR/AFT supports blockification, and we need to blockify KMeans in the future.
 # *Standardization (virutal centering)*
 ** Existing impl of standardization in linear models does NOT center the vectors by removing the means, for the purpose of keeping dataset _*sparsity*_. However, this will cause feature values with small var be scaled to large values, and underlying solver like LBFGS can not efficiently handle this case. see SPARK-34448 for details.
 ** If internal vectors are centered (like famous GLMNET), the convergence ratio will be better. In the case in SPARK-34448, the number of iteration to convergence will be reduced from 93 to 6. Moreover, the final solution is much more close to the one in GLMNET.
 ** Luckily, we find a new way to _*virtually*_ center the vectors without densifying the dataset. Good results had been observed in LoR, we will take it into account in other linear models.
 # _*Initialization (To be discussed)*_
 ** Initializing model coef with a given model, should be beneficial to: 1, convergence ratio (should reduce number of iterations); 2, model stability (may obtain a new solution more close to the previous one);
 # _*Early Stopping* *(To be discussed)*_
 ** we can compute the test error in the procedure (like tree models), and stop the training procedure if test error begin to increase;

 

      If you want to add other features in these models, please comment in the ticket.
Bug: Fix issue when compare string and IntegerType
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Jan/20 11:33
Affects Version/s: 3.0.0

When we comparing a String Type and IntegerType:
'2147483648'(StringType, which exceed Int.MaxValue) > 0(IntegerType).

Now the result of findCommonTypeForBinaryComparison(StringType, IntegerType) is IntegerType.

But the value of string may exceed Int.MaxValue, then the result is corruputed.


For example:
{code:java}
// Some comments here
CREATE TEMPORARY VIEW ta AS SELECT * FROM VALUES(CAST ('2147483648' AS STRING))  AS ta(id);
SELECT * FROM ta WHERE id > 0; // result is null
{code}
Bug: Test failure in SQL module on ppc64le
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 01/Jan/20 04:59
Affects Version/s: 2.4.0

I have been trying to build the Apache Spark on rhel_7.6/ppc64le; however, the test cases are failing in SQL module with following error :

{code}
- CREATE TABLE USING AS SELECT based on the file without write permission *** FAILED ***
  Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown (CreateTableAsSelectSuite.scala:92)

- create a table, drop it and create another one with the same name *** FAILED ***
  org.apache.spark.sql.AnalysisException: Table default.jsonTable already exists. You need to drop it first.;
at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:159)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:115)
  at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:195)
  at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:195)
  at org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3365)
  at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
{code}

Would like some help on understanding the cause for the same . I am running it on a High end VM with good connectivity.
Bug: spark.sql.hive.caseSensitiveInferenceMode does not work with Hive
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 18/Jan/20 12:08
Affects Version/s: 2.4.4

In Spark SQL, spark.sql.hive.caseSensitiveInferenceMode INFER_ONLY and INFER_AND_SAVE do not work as intended. They were supposed to infer a case-sensitive schema from the underlying files, but they do not work.
 # INFER_ONLY never works: it will always user lowercase column names from Hive metastore schema
 # INFER_AND_SAVE only works the second time {{spark.sql("SELECT …")}} is called (the first time it writes the schema to TBLPROPERTIES in the metastore and subsequent calls read that schema, so they do work)

h3. Expected behavior (according to SPARK-19611)

INFER_ONLY - infer the schema from the underlying files

INFER_AND_SAVE - infer the schema from the underlying files, save it to the metastore, and read it from the metastore on any subsequent calls
h2. Reproduce
h3. Prepare the data
h4. 1) Create a Parquet file
{code:scala}
scala> List(("a", 1), ("b", 2)).toDF("theString", "theNumber").write.parquet("hdfs:///t"){code}
 
h4. 2) Inspect the Parquet files
{code:sh}
$ hadoop jar parquet-tools-1.11.0.jar cat -j hdfs:///t/part-00000-….snappy.parquet
{"theString":"a","theNumber":1}
$ hadoop jar parquet-tools-1.11.0.jar cat -j hdfs:///t/part-00001-….snappy.parquet
{"theString":"b","theNumber":2}{code}
We see that they are saved with camelCase column names.
h4. 3) Create a Hive table 
{code:sql}
hive> CREATE EXTERNAL TABLE t(theString string, theNumber int)
 > ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
 > STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
 > OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
 > LOCATION 'hdfs:///t';{code}
 
h3. Reproduce INFER_ONLY bug
h4. 3) Read the table in Spark using INFER_ONLY
{code:sh}
$ spark-shell --master local[*] --conf spark.sql.hive.caseSensitiveInferenceMode=INFER_ONLY{code}
{code:scala}
scala> spark.sql("SELECT * FROM default.t").columns.foreach(println)
thestring
thenumber
{code}
h4. Conclusion

When INFER_ONLY is set, column names are lowercase always.
h3. Reproduce INFER_AND_SAVE bug
h4. 1) Run the for first time
{code:sh}
$ spark-shell --master local[*] --conf spark.sql.hive.caseSensitiveInferenceMode=INFER_AND_SAVE{code}
{code:scala}
scala> spark.sql("SELECT * FROM default.t").columns.foreach(println)
thestring
thenumber{code}
We see that column names are lowercase
h4. 2) Run for the second time
{code:scala}
scala> spark.sql("SELECT * FROM default.t").columns.foreach(println)
theString
theNumber{code}
We see that the column names are camelCase
h4. Conclusion

When INFER_AND_SAVE is set, column names are lowercase on first call and camelCase on subsquent calls.
Bug: More KeyValueGroupedDataset methods should be composable
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 10/Jan/20 03:24
Affects Version/s: 2.4.4

Right now many `KeyValueGroupedDataset` do not return a `KeyValueGroupedDataset`. In some cases this means we have to do multiple `groupByKey`s into order to express certain patterns.


Setup

{code:scala}
def f: T => K
def g: U => K
def h: V => K
val ds1: Dataset[T] = ???
val ds2: Dataset[U] = ???
val ds3: Dataset[V] = ??? 
val kvDs1: KeyValueGroupedDataset[K, T] = ds1.groupByKey(f)
val kvDs2: KeyValueGroupedDataset[K, U] = ds2.groupByKey(g)
val kvDs3: KeyValueGroupedDataset[K, V] = ds3.groupByKey(h)
{code}

Example one: Combining multiple CoGrouped Dataset. 

{code:scala}
// Current
kvDs1
  .coGroup(kvDs2)(k: K, it1: Iterator[T], it2: Iterator[U]) => ???: X)
  .groupByKey((x: X) => ???: K)
  .coGroup(kvDs3)(k: K, it1: Iterator[X], it2: Iterator[Y]) => ???: Z)

// Wanted
trait KeyValueGroupedDataset[K, T] {
  def coGroupKeyValueGroupedDataset[U, X](r: KeyValueGroupedDataset)(K, Iterator[T], Iterator[U] => X): KeyValueGroupedDataset[K, X]
}

kvDs1
  .coGroupKeyValueGroupedDataset(kvDs2)(k: K, it1: Iterator[T], it2: Iterator[U]) => ???: X)
  .coGroupKeyValueGroupedDataset(kvDs3)(k: K, it1: Iterator[X], it2: Iterator[Y]) => ???: Z)
{code}

Example two: Combining a reduceGroups with a coGroup 
{code:scala}
// current
val newDs1: Dataset[X] = kvDs1
  .reduceGroups((l: T, r: T) => ???: T))
  .groupByKey {case (k, _) => k }
  .mapValues { case (_, v) => v }
  .coGroup(kvDs2)(k: K, it1: Iterator[T], it2: Iterator[U]) => ???: X)

// wanted
trait KeyValueGroupedDataset[K, T] {
  def reduceGroupsKeyValueGroupedDataset(v: (V, V) => V): KeyValueGroupedDataset[K, V]
}

val newDs2: Dataset[X] = kvDs1
  .reduceGroupsKeyValueGroupedDataset((l: T, r: T) => ???: T))
  .coGroup(kvDs2)(k: K, it1: Iterator[T], it2: Iterator[U]) => ???: X)
{code}

In both cases not only are the ergonomics better, Spark will better able to optimize the code. 

For almost every method of `KeyValueGroupedDataset` we should have a matching method that returns a `KeyValueGroupedDataset`. 

We can also add a `.toDs` method which converts a `KeyValueGroupedDataset[K, V]` to a `Dataset[(K, V)]`
Bug: Introducing get_dummies method in pyspark
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 07/Jan/20 17:36
Affects Version/s: 2.4.4

Introducing get_dummies method in pyspark same as pandas.

Many times when using categorical variable and we want to flatten the data to do one-hot encoding to generate columns and fill the matrix, get_dummies is very useful in that scenario.

 

The objective here is to introduce get_dummies to pyspark.