av9ash/GitBugs huggingface dataset:
Bug: Remove the ANSI config suggestion in ARITHMETIC_OVERFLOW
Status: Open | Priority: Major | Resolution: 
Created: 13/Sep/24 10:08
Affects Version/s: 4.0.0

ARITHMETIC_OVERFLOW contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try_*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.
Bug: OptimizeCsvJsonExpr should not change the schema of underlying StructType in GetArrayStructFields
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/24 18:53
Affects Version/s: 3.5.2

The `OptimizeCsvJsonExprs` rule can potentially change the schema of the underlying `StructField` if there are differences in the field used to access the struct vs the field in the underlying struct.

This surfaces as a correctness issue where instead of picking the values for the corresponding column we end up returning NULL.

 

A simple example query is:
{code:java}
SELECT
  from_json('[{"a": '||id||', "b": '|| (2*id) ||'}]', 'array<struct<a: INT, b: INT>>').a,
  from_json('[{"a": '||id||', "b": '|| (2*id) ||'}]', 'array<struct<a: INT, b: INT>>').A
FROM
  range(3) as t{code}
 

 

Here, the result is `[0], [1], [2]` for `a` but `[null], [null], [null]` for `A`. Since struct field accessor is case-insensitive, the result should had been `[0], [1], [2]` for both.
Bug: Improve error messages from the make_timestamp() function
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 30/Sep/24 12:41
Affects Version/s: 4.0.0

Improve the error message for wrong time zone:

{code}
spark-sql (default)> select make_timestamp(1, 2, 28, 23, 1, 1, -100);
24/09/23 19:35:44 ERROR SparkSQLDriver: Failed in [select make_timestamp(1, 2, 28, 23, 1, 1, -100)]
java.time.DateTimeException: Invalid ID for ZoneOffset, invalid format: -100
	at java.base/java.time.ZoneOffset.of(ZoneOffset.java:243)
	at java.base/java.time.ZoneId.of(ZoneId.java:404)
{code}

and make_timestamp() says that valid range is 0-59 but accepts 60. Need to adjust the error message:

{code}
spark-sql (default)> select make_timestamp(1, 2, 28, 23, 1, 60);
0001-02-28 23:02:00
Time taken: 0.017 seconds, Fetched 1 row(s)
spark-sql (default)> select make_timestamp(1, 2, 28, 23, 1, 61);
Invalid value for SecondOfMinute (valid values 0 - 59): 61.
{code}
Bug: Disable CollapseProject for correlated subqueries in projection over aggregate correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/24 18:12
Affects Version/s: 4.0.0

CollapseProject should block collapsing with an aggregate if any correlated subquery is present. There are other correlated subqueries that are not ScalarSubquery that are not accounted for here.
Bug: Spark 3.5.3 does not compile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/24 13:59
Affects Version/s: 3.5.3

Issue :

When I build spark with the command
{noformat}
./dev/make-distribution.sh --name custom-spark -Phive -Phive-thriftserver -Pkubernetes
{noformat}
I have the following error :
{noformat}
[ERROR] [Error] /spark/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala:570: not found: value CatalogManager
[ERROR] [Error] /spark/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala:571: not found: type DelegatingCatalogExtension
{noformat}

Solution :

Align the 3.5.3 tag with all commit attached to 3.5.3-rc3 tag. The commit ba374c673634511373af551ba55691debd35904f which fix the build issue above is present in the 3.5.3-rc3 tag history (https://github.com/apache/spark/commits/v3.5.3-rc3/) but missing in the 3.5.3 tag history (https://github.com/apache/spark/commits/v3.5.3/).
Bug: endswith expression not working with UTF8_LCASE collation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/24 15:52
Affects Version/s: 3.4.3

 
{code:java}
select endswith('İo' collate utf8_lcase, 'İo' collate utf8_lcase)
{code}
returns 
{code:java}
false{code}
but should be 
{code:java}
true{code}
Bug: Simplify and clarify dbOptions for RocksDB
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Sep/24 21:35
Affects Version/s: 3.4.3

RocksDB.scala has 2 sets of options: columnFamilyOptions and dbOptions. columnFamilyOptions is vestigial (it is not used anywhere else in the code other than the creation of dbOptions). To simplify the code, we should remove this and set configs in dbOptions directly. Furthermore, we should rename dbOptions to rocksDbOptions for clarity.
Bug: Data races detected by SparkConnectServiceSuite over ExecuteHolder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/24 15:47
Affects Version/s: 4.0.0

1. Test issue.

https://github.com/longvu-db/spark/actions/runs/10903815379/job/30258793280

2. Real issue.

org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to eventually never returned normally. Attempted 654 times over 10.001713506999998 seconds. Last failure message: Canceled did not equal Closed Execution has not been completed in Timeout(Span(10, Seconds)).

Data race among ExecuteThreadRunner, and interrupt.
1. Thread 1 starts an ExecuteHolder.
2. Thread 2 interrupts the ExecuteHolder, before 1 is completed.
3. The ExecuteHolder never reaches 'closed'.
=> Confirmed, reproduced.
Bug: Correct handling of SPARK_USER env variable override in app master
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/24 20:49
Affects Version/s: 3.4.3

The default {{SPARK_USER}} is set to the Hadoop user in the app master container. If a user supplies a configuration value for {{{}spark.yarn.appMasterEnv.SPARK_USER{}}}, then the result gets appended to this default value like a classpath entry instead of replacing it.
Bug: How to handling Task Timeouts and Placeholder Allocation in Spark Shuffle Write Phase
Status: Open | Priority: Major | Resolution: 
Created: 24/Sep/24 03:36
Affects Version/s: 2.4.1

During the Spark shuffle write phase, the driver initiates a task to write a partition and has allocated a placeholder for the commit to that task. However, when dealing with a large volume of data, the task may fail to complete the commit task due to network issues or disk failures. In such cases, how should the driver detect the task timeout and launch a new task to commit the task for the same partition? Additionally, starting a new task raises the following issues: 1.Since the placeholder is occupied by the old task, the new task cannot obtain the placeholder for submission. How should the new task be allocated a placeholder? 2.How can the old task exit safely to ensure it does not commit the same data as the new task?

The commit protocol is the 2PC. The main process is placeholder->move->commit, [github.com/apache/spark/blob/master/core/src/main/scala/org/…|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala] here is the specific implementation.
And the commit algorithm I used is v2.

I tried searching in the history but could not find a solution to this problem. I look forward to discussing this issue with community members.
Bug: Fix return type nullability of schema_of_json
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 23/Sep/24 22:32
Affects Version/s: 3.5.2

The `schema_of_json` expression incorrectly marks its return type as non-nullable, which means that it should return a non-null result even for null input. This is not true because the expression is actually null-intolerant. This error can lead to NPE in the generated code.
Bug: Allow adding Scala UDFs in UDFRegistration
Status: Open | Priority: Major | Resolution: 
Created: 23/Sep/24 15:42
Affects Version/s: 4.0.0

In https://issues.apache.org/jira/browse/SPARK-49122 and its corresponding PR [https://github.com/apache/spark/pull/48120,] we added a new user-facing API `UDFRegistration.registerJava`. However this one only loads classes in the Java way. We would like to have another `registerScala` method to allow class loading in the Scala way,
Bug: Support single joins for full scalar subquery decorrelation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/24 22:10
Affects Version/s: 4.0.0

We should introduce Single join operator to fully support scalar subqueries.

The operator would ensure that the scalar subquery returns at most 1 row and issue a runtime error otherwise.

Design doc: https://docs.google.com/document/d/1NTsvtBTB9XvvyRvH62QzWIZuw4hXktALUG1fBP7ha1Q/edit
Bug: Add spark.shuffle.accurateBlockSkewedFactor to OSS Apache Spark configuration docs page
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/24 18:09
Affects Version/s: 3.3.0

spark.shuffle.accurateBlockSkewedFactor was added in Spark 3.3.0 in https://issues.apache.org/jira/browse/SPARK-36967 and is a useful shuffle configuration to prevent issues where HighlyCompressedMapStatus wrongly estimates the shuffle block sizes when the size distribution is skewed. Currently this config is not discoverable because it's not on the Spark config docs page and we should add it there.
Bug: Enhance "SparkConnectGrpcException: Received message larger than max" error message
Status: Open | Priority: Major | Resolution: 
Created: 20/Sep/24 07:26
Affects Version/s: 4.0.0

SparkConnectGrpcException: Received message larger than max
=> Users have no idea what this means, how to lift the limit.

Need to be more descriptive.
Bug: Make `spark-docker` repository's Dockerfiles up-to-date
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/24 22:03
Affects Version/s: 4.0.0

Apache Spark Docker repository is outdated. We need to update it with the latest change of Apache Spark repository.
 * [https://github.com/apache/spark-docker]

 

Since the docker files are different, the implementations are different for the same goals.
Bug: Remove Self Type for Dataset
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/24 00:09
Affects Version/s: 4.0.0

We are currently using a self type for Dataset. This makes it a bit easier (in Scala land) to deal with covariant parameters and return types.

The current approach is unfortunately a bit leaky. For example the self-type leaks into SparkSession and things like the Catalog. This is a bit distracting especially when you use these classes in java.

The proposal is to either remove the self-type and rely on overrides (which have to anyway), or to take another stab at using type members.
Bug: Increase maxBatchSize for Connect's sqlCommandResult
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/24 15:29
Affects Version/s: 4.0.0

Increase the default maxBatchSize from 4MiB * 0.7 to 128MiB (=
CONNECT_GRPC_MAX_MESSAGE_SIZE) * 0.7 when creating the single Arrow batch for the SqlCommandResult in the SparkConnectPlanner. This lets us return much larger LocalRelations in the SqlCommandResult (for example for the SHOW PARTITIONS command) while still staying within the GRPC message size limit.
Bug: Support `spark.test.master` in `SparkSubmitArguments`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/24 20:50
Affects Version/s: 4.0.0

To allow users to control the default master setting during testing and documentation generation.

BEFORE (`local[*]`)
{code:java}
$ bin/pyspark
Python 3.9.19 (main, Jun 17 2024, 15:39:29)
[Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-pattern-layout-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/09/16 13:53:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / _/_  ___ ____/ /_
    \ \/ _ \/ _ `/ __/  '/
   /__ / ._/_,// //_\   version 4.0.0-SNAPSHOT
      /_/
Using Python version 3.9.19 (main, Jun 17 2024 15:39:29)
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1726519982935).
SparkSession available as 'spark'.
>>>{code}
 

AFTER (`local[1]`)
{code:java}
 $ JDK_JAVA_OPTIONS="-Dspark.test.master=local[1]" bin/pyspark
NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
Python 3.9.19 (main, Jun 17 2024, 15:39:29)
[Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-pattern-layout-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/09/16 13:51:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / _/_  ___ ____/ /_
    \ \/ _ \/ _ `/ __/  '/
   /__ / ._/_,// //_\   version 4.0.0-SNAPSHOT
      /_/
Using Python version 3.9.19 (main, Jun 17 2024 15:39:29)
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[1], app id = local-1726519863363).
SparkSession available as 'spark'.
>>>{code}
Bug: Stop ignoring auth errors when ignoreCorruptFiles=1 in FileScanRDD
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/24 17:17
Affects Version/s: 4.0.0

We currently lose data when the ignoreCorruptFiles flag is set to true because file Authorization Errors are handled as if they were corrupt files. We ignore these files despite them having no corrupt records. We’d like to fix this bug by failing the stream when authorization errors are found.

To prevent regressions we will add a SQLConf flag that can be temporarily set to revert to previous behavior. The idea is to deprecate the flag at a later major version release.
Bug: Don't use deprecated Hive JDBC client parameters in docs / tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/24 23:27
Affects Version/s: 4.0.0

While looking at some hive-thriftserver unit tests logs, I saw spam like
{code:java}
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: ***** JDBC param deprecation *****
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: The use of hive.server2.transport.mode is deprecated.
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: Please use transportMode like so: jdbc:hive2://<host>:<port>/dbName;transportMode=<transport_mode_value>
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: ***** JDBC param deprecation *****
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: The use of hive.server2.thrift.http.path is deprecated.
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: Please use httpPath like so: jdbc:hive2://<host>:<port>/dbName;httpPath=<http_path_value> {code}
from deprecated JDBC connection string parameter usage.

It looks like we're using / recommending parameters that have been deprecated for nearly a decade since https://issues.apache.org/jira/browse/HIVE-6972 

We can clean these up to reduce some log spam for our tests (and our end users, if we update the docs).
Bug: Scalar subquery inside VALUES clause produces INTERNAL_ERROR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/24 08:49
Affects Version/s: 4.0.0

Scalar subquery inside VALUES clause produces INTERNAL_ERROR:


`org.apache.spark.SparkException: [INTERNAL_ERROR] Found the unresolved operator: 'Project [unresolvedalias(1)] SQLSTATE: XX000`


It would be better to change it to a nice user-facing error.
Bug: Two instances of Spark Operator running at the same time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/24 06:21
Affects Version/s: 4.0.0

There is one scenario, in a cluster, user deployed multi instances of operator and make them watching on several namespaces. For example,

operator A in default namespace, watching on namespace spark-1
operator B in default-2 namespace, watching on namespace spark-2..
Bug: Add DataFrame.map equivalent to RDD.map
Status: Open | Priority: Major | Resolution: 
Created: 16/Sep/24 23:50
Affects Version/s: 3.5.2

Lots of reasons to migrate RDD.map() to DataFrame.map(), especially required with Spark Connect. We see lots of users with 97% DataFrame code but occasional use of RDD.map which kills their ability to adopt the latest versions.

Create a compatibility api to make RDD to DataFrame migration easier.
 
Here is a prototype:
{code:python}
## Setup convenience function for DataFrame.map equivalent to rdd.map
from pyspark.sql.connect.dataframe import DataFrame
from pyspark.sql.functions import udf, struct
from pyspark.sql.types import StringType, StructType

class RddLike:
   def __init__(self, generator):
       self.gen = generator
   def collect(self):
       return [val[0] for val in self.gen.collect()]

def map(self, func, returnType=StringType()):
    return RddLike(self.select(udf(func, returnType, useArrow=True)(struct("*"))))

DataFrame.map = map

##
df = spark.createDataFrame([("Alice", 100), ("Bob", 200)], ["name", "age"])
df.map(lambda row: f"{row.name=} {row.age=}").collect()
{code}
Bug: Unable to use alias with LATERAL VIEW in Spark SQL
Status: Open | Priority: Major | Resolution: 
Created: 16/Sep/24 16:33
Affects Version/s: 3.5.2

Using Spark SQL with a lateral view as a source and an alias in the projection, spark fails to parse the SQL:
{code:java}
scala> spark.sql("SELECT arr as c FROM LATERAL VIEW OUTER INLINE(ARRAYS_ZIP(ARRAY(1,3))) `t` as `arr`").show
org.apache.spark.sql.catalyst.parser.ParseException:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'INLINE'.(line 1, pos 54)== SQL ==
SELECT ELEMENT_AT(arr,1) as c FROM LATERAL VIEW OUTER INLINE(ARRAYS_ZIP(ARRAY(ARRAY(1,2),ARRAY(3,4)))) `t` as `arr`
------------------------------------------------------^^^  at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(parsers.scala:257)
  at org.apache.spark.sql.catalyst.parser.AbstractParser.parse(parsers.scala:98)
  at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:54)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(AbstractSqlParser.scala:68)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$5(SparkSession.scala:684)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:138)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$4(SparkSession.scala:683)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:682)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:713)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:744)
  ... 47 elided {code}
without the alias, the query works fine:
{code:java}
scala> spark.sql("SELECT arr FROM LATERAL VIEW OUTER INLINE(ARRAYS_ZIP(ARRAY(1,3))) `t` as `arr`").show
+----+
|FROM|
+----+
|   1|
|   3|
+----+ {code}
side note: the assigned alias in the results is FROM, which doesn't seem right.

It seems the SQL is not being parsed correctly in this case.
Bug: Cached local relation lookup failure
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 16/Sep/24 10:59
Affects Version/s: 4.0.0

```
  private def transformCachedLocalRelation(rel: proto.CachedLocalRelation): LogicalPlan = {
    val blockManager = session.sparkContext.env.blockManager
    val blockId = CacheId(sessionHolder.session.sessionUUID, rel.getHash)
    val bytes = blockManager.getLocalBytes(blockId)
    bytes
      .map { blockData =>
        try {
          val localRelation = proto.Relation
            .newBuilder()
            .getLocalRelation
            .getParserForType
            .parseFrom(blockData.toInputStream())
          transformLocalRelation(localRelation)
        } finally {
          blockManager.releaseLock(blockId)
        }
      }
      .getOrElse {
        throw InvalidPlanInput(
          s"Not found any cached local relation with the hash: ${blockId.hash} in " +
            s"the session with sessionUUID ${blockId.sessionUUID}.")
      }
  }
```

=> `blockManager.releaseLock(blockId)` is wrong because locks are released inside `val bytes = blockManager.getLocalBytes(blockId)`.
Bug: WholeStageCodegen lasts up to 47min
Status: Open | Priority: Major | Resolution: 
Created: 14/Sep/24 19:45
Affects Version/s: 3.4.1

Using Spark 3.4.1 on EMR 6.15.0 and I've noticed that job is running long while all jobs and stages are showing small times <1min but the SQL/DataFrame  would show Completed Queries and I can find that there is a query that runs long and the issue is that WholeStageCodegen can last up to 47min.  The step was Project step and min time is 146ms, med is 232ms and max is 47.2min. 
While I can do a rewrite of the logic (it is group by with 6 min/max aggregation and 1 collect_list) I was surprised to learn that there is no timeout to take out executor that is spending 47min in WholeStageCodegen stage. 
One ask would be to have a threshold for how long WholeStageCodegen can last, say 500ms and another ask is to automatically detect long lasting WholeStageCodegen steps
Bug: Pick up system property defaults in SharedSparkContext test suite mixin
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/24 19:09
Affects Version/s: 4.0.0

The SharedSparkContext test suite mixin currently instantiates a SparkConf with `new SparkConf(false)`, which does not pick up defaults from system properties and thus misses certain test-default configurations and settings which are specified via test runner options in SBT or Maven; see [https://github.com/apache/spark/blob/08a26bb56cfb48f27c68a79be1e15bc4c9e466e0/core/src/test/scala/org/apache/spark/SharedSparkContext.scala#L30] and [https://github.com/apache/spark/blob/08a26bb56cfb48f27c68a79be1e15bc4c9e466e0/project/SparkBuild.scala#L1616-L1633] .

As a consequence, a number of Spark's test suites are missing test-only defaults and this can adversely impact the performance and flakiness of those tests.

I think that we should update SharedSparkContext to do `new SparkConf(true)` (a.k.a `new SparkConf()`) to address this problem. Note that most of our other test suites and mxins are already doing this, including SharedSparkSession in SparkSQL.
Bug: Remove the ANSI config suggestion in DATATYPE_MISMATCH.CAST_WITH_CONF_SUGGESTION
Status: Open | Priority: Major | Resolution: 
Created: 13/Sep/24 10:21
Affects Version/s: 4.0.0

DATATYPE_MISMATCH.CAST_WITH_CONF_SUGGESTION contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try_*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.
Bug: Remove the ANSI config suggestion in DIVIDE_BY_ZERO
Status: Open | Priority: Major | Resolution: 
Created: 13/Sep/24 10:17
Affects Version/s: 4.0.0

DIVIDE_BY_ZERO contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try_*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.