av9ash/GitBugs huggingface dataset:
Bug: `Cancel workflow` can not cancel some jobs
Status: Open | Priority: Minor | Resolution: 
Created: 09/Aug/22 14:43
Affects Version/s: 3.4.0

I have been observing this behavior for a long time:

sometime I want to cancel the workflow to release resource for other workflows, so I click the {{*{color:red}Cancel workflow{color}*}} button, most jobs in current workflow will be canceled in several minutes, but some jobs can never be killed, including:

* Run / Build modules: pyspark-sql, pyspark-mllib, pyspark-resource 
* Run / Build modules: pyspark-core, pyspark-streaming, pyspark-ml 
* Run / Build modules: pyspark-pandas 
* Run / Build modules: pyspark-pandas-slow 
* Run / Linters, licenses, dependencies and documentation generation
Bug: YarnClusterSuite should not ABORTED when there is no Python3 environment
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Aug/22 17:19
Affects Version/s: 3.4.0

Execute "build/mvn clean test -pl resource-managers/yarn -Pyarn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite -am" when the python 3 running environment is not configured:
{code:java}
YarnClusterSuite:
org.apache.spark.deploy.yarn.YarnClusterSuite *** ABORTED ***
  java.lang.RuntimeException: Unable to load a Suite class that was discovered in the runpath: org.apache.spark.deploy.yarn.YarnClusterSuite
  at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:81)
  at org.scalatest.tools.DiscoverySuite.$anonfun$nestedSuites$1(DiscoverySuite.scala:38)
  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
  at scala.collection.Iterator.foreach(Iterator.scala:943)
  at scala.collection.Iterator.foreach$(Iterator.scala:943)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  at scala.collection.IterableLike.foreach(IterableLike.scala:74)
  at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
  at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
  at scala.collection.TraversableLike.map(TraversableLike.scala:286)
  ...
Run completed in 833 milliseconds.
Total number of tests run: 0
Suites: completed 1, aborted 1
Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
*** 1 SUITE ABORTED ***
 {code}
Bug: Avoid noisy warnings logs when spark.scheduler.listenerbus.metrics.maxListenerClassesTimed = 0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Aug/22 01:55
Affects Version/s: 3.4.0

If {{spark.scheduler.listenerbus.metrics.maxListenerClassesTimed}} has been set to {{0}} to disable listener timers then listener registration will trigger noisy warnings like
{code:java}
LiveListenerBusMetrics: Not measuring processing time for listener class org.apache.spark.sql.util.ExecutionListenerBus because a maximum of 0 listener classes are already timed.{code}
warnings.

We should change the code to not print this warning when maxListenerClassesTimed = 0.

I don't plan to work on this myself.
Bug: HiveExternalCatalogVersionsSuite will test all spark versions and aborted when Python 2.7 is used 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Aug/22 04:59
Affects Version/s: 3.4.0

When the test environment is java 8 + Python 2.7,  HiveExternalCatalogVersionsSuite will test all Spark 3.x versions and Spark 2.4.8,  and all test will ABORTED
Bug: The result is strange when casting string to date in ORC reading via Schema Evolution
Status: Open | Priority: Minor | Resolution: 
Created: 31/Aug/22 08:31
Affects Version/s: 3.1.1

I created an ORC file by the code as follows.
{code:java}
val data = Seq(
    ("", "2022-01-32"),  // pay attention to this, null
    ("", "9808-02-30"),  // pay attention to this, 9808-02-29
    ("", "2022-06-31"),  // pay attention to this, 2022-06-30
)
val cols = Seq("str", "date_str")
val df=spark.createDataFrame(data).toDF(cols:_*).repartition(1)
df.printSchema()
df.show(100)
df.write.mode("overwrite").orc("/tmp/orc/data.orc")
{code}
Please note that these three cases are invalid date.
And I read it via:
{code:java}
scala> var df = spark.read.schema("date_str date").orc("/tmp/orc/data.orc"); df.show()
+----------+
|  date_str|
+----------+
|      null|
|9808-02-29|
|2022-06-30|
+----------+{code}
Why is `2022-01-32` converted to `null`, while `9808-02-30` is converted to `9808-02-29`?

Intuitively, they are invalid date, we should return 3 nulls. Is it a bug or a feature?

 

 

*Background*
 * I am working on the project: [https://github.com/NVIDIA/spark-rapids]
 * And I am working on a feature, that is to support reading ORC file as an cuDF (CUDA DataFrame). cuDF is an in-memory data-format of GPU.
 * I need to follow the behaviors of ORC reading in CPU. Otherwise, the users of spark-rapids will feel strange with the results.
 * Therefore I want to know why those happpened.
Bug: Temporarily disable flaky test cases in ImageFileFormatSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Aug/22 03:41
Affects Version/s: 3.2.2

3 test cases in ImageFileFormatSuite become flaky in the GitHub action tests:

[https://github.com/apache/spark/runs/7941765326?check_suite_focus=true]

Before they are fixed(https://issues.apache.org/jira/browse/SPARK-40171), I suggest disabling them in OSS.
Bug: Corrected  console output in quick-start -  Datatype Integer instead of Long
Status: Closed | Priority: Minor | Resolution: Fixed
Created: 30/Aug/22 02:15
Affects Version/s: 3.1.2

h3. What changes were proposed in this pull request?

Corrected datatype output of command from Long to Int
h3. Why are the changes needed?

It shows incorrect datatype
h3. Does this PR introduce _any_ user-facing change?

Yes. It proposes changes in documentation for console output.
[!https://user-images.githubusercontent.com/12110063/187332894-af2b6b43-7ff3-4062-8370-de4b477f178b.png!|https://user-images.githubusercontent.com/12110063/187332894-af2b6b43-7ff3-4062-8370-de4b477f178b.png]
h3. How was this patch tested?

Manually checked the changes by previewing markdown output. I tested output by installing spark 3.3.0 locally and running commands present in quick start docs
Bug: Make compute.max_rows as None working in DataFrame.style
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Aug/22 05:34
Affects Version/s: 3.4.0

{code}
import pyspark.pandas as ps
ps.set_option("compute.max_rows", None)
ps.get_option("compute.max_rows")
ps.range(1).style
{code}

fails as below:

{code}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../spark/python/pyspark/pandas/frame.py", line 3656, in style
    pdf = self.head(max_results + 1)._to_internal_pandas()
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
{code}
Bug: Remove since usage in streaming/query.py and window.py
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Aug/22 11:41
Affects Version/s: 3.4.0

NumPy documentation style doesn't play well with our pyspark.since (because it just adds versionadded directive in the end but NumPy documentation assume to put these in the corresponding section).

We should better remove these usages out.
Bug: Support multiple Column drop in R
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Aug/22 17:19
Affects Version/s: 3.3.0

This is a followup on SPARK-39895. The PR previously attempted to adjust implementation for R as well to match signatures but that part was removed and we only focused on getting python implementation to behave correctly.

*{{Change supports following operations:}}*

{{df <- select(read.json(jsonPath), "name", "age")}}

{{df$age2 <- df$age}}

{{df1 <- drop(df, df$age, df$name)}}
{{expect_equal(columns(df1), c("age2"))}}

{{df1 <- drop(df, df$age, column("random"))}}
{{expect_equal(columns(df1), c("name", "age2"))}}

{{df1 <- drop(df, df$age, df$name)}}
{{expect_equal(columns(df1), c("age2"))}}
Bug: Exclude hive-vector-code-gen dependency
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Aug/22 06:05
Affects Version/s: 3.4.0

hive-exec depends on importing velocity. Velocity has an older version and has many security issues

https://issues.apache.org/jira/browse/HIVE-25726

 

!image-2022-08-02-14-05-55-756.png!
Bug: Task listener overloads no longer needed with JDK 8+
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 18/Aug/22 22:59
Affects Version/s: 3.3.0

TaskContext defines methods for registering completion and failure listeners, and the respective listener types qualify as functional interfaces in JDK 8+. This leads to awkward ambiguous overload errors with the overload of each function, that takes a function directly instead of a listener. Now that JDK 8 is the minimum allowed, we can remove the unnecessary overloads, which not only simplifies the code, but also removes a source of frustration since it can be nearly impossible to predict when an ambiguous overload might be triggered.
Bug: Converting Tuple2 to Scala Map via `.toMap` is slow
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Aug/22 06:57
Affects Version/s: 3.1.2

Converting Tuple2 to Scala Map via `.toMap` is slow

!image-2022-08-22-14-58-53-046.png!

!image-2022-08-22-14-58-26-491.png!
Bug: Make the V2 table error message more meaningful
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Aug/22 18:16
Affects Version/s: 3.4.0

When V2 catalog is not configured, Spark fails to access/create a table using the V2 API and silently falls back to attempting to do the same operation using the V1 Api. This happens frequently among the users. We want to have a better error message so that users can fix the configuration/usage issue by themselves.
Bug: Upgrade postgresql to 42.5.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Aug/22 10:44
Affects Version/s: 3.4.0

Security
- fix: [CVE-2022-31197|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-31197] Fixes SQL generated in PgResultSet.refresh() to escape column identifiers so as to prevent SQL injection.
  - Previously, the column names for both key and data columns in the table were copied as-is into the generated
  SQL. This allowed a malicious table with column names that include statement terminator to be parsed and
  executed as multiple separate commands.
  - Also adds a new test class ResultSetRefreshTest to verify this change.
  - Reported by [Sho Kato](https://github.com/kato-sho)

[Release note|https://github.com/pgjdbc/pgjdbc/commit/bd91c4cc76cdfc1ffd0322be80c85ddfe08a38c2]
Bug: After `RemoveRedundantAggregates`, `PullOutGroupingExpressions` should applied to avoid attribute missing when use complex expression.
Status: In Progress | Priority: Minor | Resolution: 
Created: 31/Aug/22 07:06
Affects Version/s: 3.2.0

{{--table}}
{{create}}  {{table}} {{miss_expr(id }}{{{}int{}}}{{{},{}}}{{{}name{}}} {{string,age }}{{{}double{}}}{{{}) stored {}}}{{as}} {{textfile}}
{{--data}}
{{insert}} {{overwrite }}{{table}} {{miss_expr }}{{{}values{}}}{{{}(1,{}}}{{{}'ox'{}}}{{{},1.0),(1,{}}}{{{}'oox'{}}}{{{},2.0),(2,{}}}{{{}'ox'{}}}{{{},3.0),(2,{}}}{{{}'xxo'{}}}{{{},4.0){}}}
{{--failure sql}}

{{select}} {{{}id,{}}}{{{}name{}}}{{{},nage {}}}{{as}} {{nnnnn }}{{{}from{}}}{{{}({}}}
{{select}} {{{}id,{}}}{{{}name{}}}{{{},if(age>3,100,200) {}}}{{as}} {{nage }}{{from}} {{miss_expr }}{{group}} {{by}} {{{}id,{}}}{{{}name{}}}{{{},age{}}}
{{) }}{{group}} {{by}} {{{}id,{}}}{{{}name{}}}{{{},nage{}}}

--error stack
{{Caused by: java.lang.IllegalStateException: Couldn't find age#4 in [id#2,name#3,if ((age#4 > 3.0)) 100 else 200#12|#2,name#3,if ((age#4 > 3.0)) 100 else 200#12]}}
{{at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:80)}}
{{at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:73)}}
{{at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)}}
Bug: Reactor ParquetScanBuilder DataSourceV2 interface implementation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Aug/22 03:51
Affects Version/s: 3.3.0

Now `FileScanBuilder` interface is not fully implemented in `ParquetScanBuilder` like `OrcScanBuilder`,`AvroScanBuilder`,`CSVScanBuilder`....
In order to unify the logic of the code and make it clearer, this part of the implementation is unified.
Bug: Upgrade netty from 4.1.77 to 4.1.80
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Aug/22 07:39
Affects Version/s: 3.4.0

* [https://netty.io/news/2022/06/14/4-1-78-Final.html]
 * [https://netty.io/news/2022/07/11/4-1-79-Final.html]
 * https://netty.io/news/2022/08/26/4-1-80-Final.html
Bug: Add adaptive plan case in ReplaceNullWithFalseInPredicateEndToEndSuite
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 12/Aug/22 00:13
Affects Version/s: 3.4.0

Currently `ReplaceNullWithFalseInPredicateEndToEndSuite` assumes that adaptive query execution is turned off. We should add cases `spark.sql.adaptive.forceApply=true`
Bug: New OFFSET clause does not use new error framework 
Status: Open | Priority: Minor | Resolution: 
Created: 24/Aug/22 16:12
Affects Version/s: 3.4.0

CREATE TEMP VIEW person (name, age)
AS VALUES ('Zen Hui', 25),
('Anil B' , 18),
('Shone S', 16),
('Mike A' , 25),
('John A' , 18),
('Jack N' , 16);

SELECT name, age FROM person ORDER BY name OFFSET length(name);

Error in query: The offset expression must evaluate to a constant value, but got length(person.name);

Offset length(name#181)

+- Sort [name#181 ASC NULLS FIRST], true

   +- Project [name#181, age#182]

      +- SubqueryAlias person

         +- View (`person`, [name#181,age#182])

            +- Project [cast(col1#183 as string) AS name#181, cast(col2#184 as int) AS age#182]

               +- LocalRelation [col1#183, col2#184|#183, col2#184]


 

Returning the plan here is quite pointless as well. The context would be more interesting.
Bug: REST API for SQL level information does not show information on running queries
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 18/Aug/22 16:12
Affects Version/s: 3.2.0

Hi All,

We noticed that the SQL information REST API implemented in https://issues.apache.org/jira/browse/SPARK-27142 does not return back SQL queries which are currently running. We can only see queries which are completed/failed.

As far as I can see, this should be supported since one of the fields in the returned JSON is "runningJobIds".
Bug: The partitionSpec should be distinct keys after filter one row of row_number
Status: In Progress | Priority: Minor | Resolution: 
Created: 21/Aug/22 09:35
Affects Version/s: 3.3.0

For query
{code:sql}
SELECT *
  FROM (
    SELECT *, row_number() over(partition by key order by value) rn
    FROM testData t
  ) t1
  WHERE rn=1
{code}
column *key* will be distinct
Bug: Upgrade RoaringBitmap from 0.9.30 to 0.9.31
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Aug/22 07:34
Affects Version/s: 3.4.0

https://github.com/RoaringBitmap/RoaringBitmap/compare/0.9.30...0.9.31

[simplify BatchIterators, fix bug in advanceIfNeeded (|https://github.com/RoaringBitmap/RoaringBitmap/commit/56b1bba400e9f91c682648fa90b890f3a0bb561c] [#573|https://github.com/RoaringBitmap/RoaringBitmap/pull/573] [)|https://github.com/RoaringBitmap/RoaringBitmap/commit/56b1bba400e9f91c682648fa90b890f3a0bb561c]
Bug: Optionally use a serialized storage level for DataFrame.localCheckpoint()
Status: Open | Priority: Minor | Resolution: 
Created: 20/Aug/22 14:12
Affects Version/s: 3.3.0

In PySpark 3.3.0 `DataFrame.localCheckpoint()` stores the RDD checkpoints using the "Disk Memory *Deserialized* 1x Replicated" storage level. Looking through the Python code and the documentation, I haven't found any possibility to change this.

As serialized RDDs are often a lot smaller than deserialized ones - I have seen examples where a 40GB deserialized RDD shrank to 200MB when serialized - I would usually like to create local checkpoints that are stored in serialized instead of deserialized format.

To make this possible, we could e.g. add an optional `storage_level` argument to `DataFrame.localCheckpoint()` similar to `DataFrame.persist()` or add a global configuration option similar to `spark.checkpoint.compress`.
Bug: Executor ConfigMap is not mounted if profile is not default
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Aug/22 22:12
Affects Version/s: 3.2.0

When executor config map is made optional in SPARK-34316, mount volume is unconditionally disabled erroneously when non-default profile is used.

When spark.kubernetes.executor.disableConfigMap is false, expected behavior is that the ConfigMap is mounted regardless of executor's resource profile. However, it is not mounted if the resource profile is non-default.
Bug: Filter jobs by Job Group in UI & API
Status: Open | Priority: Minor | Resolution: 
Created: 18/Aug/22 15:10
Affects Version/s: 3.2.0

Hi all,

We have some cases where it would be useful to be able to have a list of all jobs belonging to the same job group in the Spark UI. This doesn't yet seem possible. We also noticed that in the REST API provided by spark, it's not possible to filter in jobs by specific job groups.

It would be great if we can have the functionality to filter on jobs based on their job group id in the UI and in the API.
Bug: MultilayerPerceptronClassifier rawPredictionCol param missing from setParams
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Aug/22 02:43
Affects Version/s: 3.3.0

https://issues.apache.org/jira/browse/SPARK-37398 inlined type hints in Pyspark ML's classification.py but inadvertently removed the parameter rawPredictionCol from MultilayerPerceptronClassifier's setParams. This causes its constructor to fail when this param is set in the constructor, as it isn't recognized by setParams, called by the constructor.
Bug: Improve repartition in ReplaceCTERefWithRepartition
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Aug/22 10:43
Affects Version/s: 3.4.0

If cte can not inlined, the ReplaceCTERefWithRepartition will add repartition to force a shuffle so that the reference can reuse shuffle exchange.
The added repartition should be optimized by AQE for better performance.

If the user has specified a rebalance, the ReplaceCTERefWithRepartition should skip add repartition.
Bug: LevelDB/RocksDBIterator.next should return false after iterator or db close
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Aug/22 16:41
Affects Version/s: 3.4.0

{code:java}
@Test
public void testHasNextAndNextAfterIteratorClose() throws Exception {
  String prefix = "test_db_iter_close.";
  String suffix = ".ldb";
  File path = File.createTempFile(prefix, suffix);
  path.delete();
  LevelDB db = new LevelDB(path);
  // Write one records for test
  db.write(createCustomType1(0));

  KVStoreIterator<CustomType1> iter =
    db.view(CustomType1.class).closeableIterator();
  // iter should be true
  assertTrue(iter.hasNext());
  // close iter
  iter.close();
  // iter.hasNext should be false after iter close
  assertFalse(iter.hasNext());
  // iter.next should throw NoSuchElementException after iter close
  assertThrows(NoSuchElementException.class, iter::next);

  db.close();
  assertTrue(path.exists());
  FileUtils.deleteQuietly(path);
  assertFalse(path.exists());
}

@Test
public void testHasNextAndNextAfterDBClose() throws Exception {
  String prefix = "test_db_db_close.";
  String suffix = ".ldb";
  File path = File.createTempFile(prefix, suffix);
  path.delete();
  LevelDB db = new LevelDB(path);
  // Write one record for test
  db.write(createCustomType1(0));

  KVStoreIterator<CustomType1> iter =
    db.view(CustomType1.class).closeableIterator();
  // iter should be true
  assertTrue(iter.hasNext());
  // close db
  db.close();
  // iter.hasNext should be false after db close
  assertFalse(iter.hasNext());
  // iter.next should throw NoSuchElementException after db close
  assertThrows(NoSuchElementException.class, iter::next);

  assertTrue(path.exists());
  FileUtils.deleteQuietly(path);
  assertFalse(path.exists());
} {code}
 

For the above two cases, when iterator/db is closed, `hasNext` will return true, and `next` will return the value not obtained before close.
Bug: MillisToTimestamp overflows on Long.MAX_VALUE
Status: Open | Priority: Minor | Resolution: 
Created: 16/Aug/22 09:23
Affects Version/s: 3.2.0

MillisToTimestamp throws an ArithmeticOverflowException on Long.MAX_VALUE even though Instant.ofEpochMillis(Long.MAX_VALUE) is valid.

The reason this happens is that MillisToTimestamp delegates to MicrosToTimestamp and so it multiplies the input with a factor and tries to fit that into a long.
Bug: Support number-only column names in ORC data sources when orc impl is hive
Status: In Progress | Priority: Minor | Resolution: 
Created: 15/Aug/22 03:29
Affects Version/s: 3.2.2

This problem is similar to SPARK-36663, both because *CatalystSqlParser.parseDataType* fails to parse if a column name (and nested field) consists of only numbers.

SPARK-36663solves the problem when configuring spark.sql.orc.impl=native (default is native), but when configuring spark.sql.orc.impl=hive, it still throws an error now
Bug: MAVEN_OPTS in make-distributions.sh is different from one specified in pom.xml
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Aug/22 10:10
Affects Version/s: 3.2.2

Building spark with make-distribution.sh is failed with default setting because default MAVEN_OPTS is different from the one in pom.xml.
 It is related to [SPARK-35825|https://issues.apache.org/jira/browse/SPARK-35825].


PR: https://github.com/apache/spark/pull/37510
Bug: Upgrade Apache ZooKeeper to 3.8.0
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 13/Aug/22 11:01
Affects Version/s: 3.4.0

SNYK want to Upgrade org.apache.zookeeper:zookeeper from 3.6.2 to 3.8.0

The recommended version is 4 versions ahead of your current version. 
The recommended version was released 5 months ago, on 2022-02-25.
Bug: Update plugins to latest versions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Aug/22 11:52
Affects Version/s: 3.4.0

include:
 # maven-compiler-plugin (from 3.8.1 to 3.10.1)
 # maven-jar-plugin (from 3.1.2 to 3.2.2)
 # maven-javadoc-plugin (from 3.1.1 to 3.4.0)
 # checkstyle (from 8.43 to 9.3)
Bug: Spark - Creating Sub Folder while writing to Partitioned Hive Table
Status: Open | Priority: Minor | Resolution: 
Created: 12/Aug/22 20:38
Affects Version/s: 2.4.7

We had been writing to a Partitioned Hive Table and realized that data is being written has sub-folder.

For ex- Refer Table definition as below - 

_Create table T1 ( name string, address string) Partitioned by (process_date string) stored as parquet location '/mytable/a/b/c/org=employee';_

 

While writing to table HDFS path being written looks something like this - 

{_}/mytable/a/b/c/org=employee/{_}{_}process_date=20220812/{_}{color:#de350b}_org=employee_{color}

 

The unnecessary addition of  _org=employee_ after process_date partition is because Hive Table has location consisting "=" operator, which Hive uses as syntax to determine partition column.

Re-defining Table resolves above problem - 

_Create table T1 ( name string, address string) Partitioned by (process_date string) stored as parquet location '/mytable/a/b/c/employee';_