av9ash/GitBugs huggingface dataset:
Bug: [DataSource V2] Documentation on SupportsReportStatistics Outdated?
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jun/20 19:31
Affects Version/s: 2.4.6

I was wondering if the documentation on SupportsReportStatistics [1][3] about its interaction with the planner and predicate pushdowns is still accurate. It says:

"Implementations that return more accurate statistics based on pushed operators will not improve query performance until the planner can push operators before getting stats."

 

Is this still accurate? When looking through the code it seems like there is now functionality that explicitly wants the operators pushed down [2]. Is the documentation for SupportsReportStatistics referring to something other than [2] or should it be updated?

 

[[1]https://spark.apache.org/docs/2.4.6/api/java/org/apache/spark/sql/sources/v2/reader/SupportsReportStatistics.html|https://spark.apache.org/docs/2.4.6/api/java/org/apache/spark/sql/sources/v2/reader/SupportsReportStatistics.html]

[2] [https://github.com/apache/spark/blob/d0800fc8e2e71a79bf0f72c3e4bc608ae34053e7/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala#L86]

[3][https://spark.apache.org/docs/3.0.0-preview/api/java/org/apache/spark/sql/connector/read/SupportsReportStatistics.html]
Bug: Fix AnalysisException messages at UNION/INTERSECT/EXCEPT/MINUS operations
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Jun/20 16:45
Affects Version/s: 2.1.3

Union and set operations can only be performed on tables with the compatible column types,while when we have more than two column, the warning messages will have wrong column index.Steps to reproduce.

Step1:prepare test data
{code:java}
drop table if exists test1; 
drop table if exists test2; 
drop table if exists test3;
create table if not exists test1(id int, age int, name timestamp);
create table if not exists test2(id int, age timestamp, name timestamp);
create table if not exists test3(id int, age int, name int);
insert into test1 select 1,2,'2020-01-01 01:01:01';
insert into test2 select 1,'2020-01-01 01:01:01','2020-01-01 01:01:01'; 
insert into test3 select 1,3,4;
{code}
Step2:do query:
{code:java}
Query1:
select * from test1 except select * from test2;
Result1:
Error: org.apache.spark.sql.AnalysisException: Except can only be performed on tables with the compatible column types. timestamp <> int at the second column of the second table;; 'Except false :- Project [id#620, age#621, name#622] : +- SubqueryAlias `default`.`test1` : +- HiveTableRelation `default`.`test1`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#620, age#621, name#622] +- Project [id#623, age#624, name#625] +- SubqueryAlias `default`.`test2` +- HiveTableRelation `default`.`test2`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#623, age#624, name#625] (state=,code=0)
Query2:
select * from test1 except select * from test3;
Result2:
Error: org.apache.spark.sql.AnalysisException: Except can only be performed on tables with the compatible column types. int <> timestamp at the 2th column of the second table;; 'Except false :- Project [id#632, age#633, name#634] : +- SubqueryAlias `default`.`test1` : +- HiveTableRelation `default`.`test1`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#632, age#633, name#634] +- Project [id#635, age#636, name#637] +- SubqueryAlias `default`.`test3` +- HiveTableRelation `default`.`test3`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#635, age#636, name#637] (state=,code=0)
{code}
the result of query1 is correct, while query2 have the wrong errors,it should be the third column

Here has the wrong column index.

+Error: org.apache.spark.sql.AnalysisException: Except can only be performed on tables with the compatible column types. int <> timestamp at the *2th* column of the second table+

We may need to change to the following

+Error: org.apache.spark.sql.AnalysisException: Except can only be performed on tables with the compatible column types. int <> timestamp at the *third* column of the second table+
Bug: Add a parameter to enable skip verify the input and output paths for InsertIntoHadoopFsRelationCommand
Status: In Progress | Priority: Minor | Resolution: 
Created: 29/Jun/20 06:52
Affects Version/s: 3.0.0

When INSERT OVERWRITE a partitioned data source table, spark will verify input and output paths by default. Set to false to skip the verify in some cases.  (eg. INSERT OVERWRITE t partition(dt=a) SELECT * FROM t WHERE dt=b)
Bug: SQL hash function handling of nulls makes collision too likely
Status: Open | Priority: Minor | Resolution: 
Created: 26/Jun/20 18:51
Affects Version/s: 3.0.0

this ticket is about org.apache.spark.sql.functions.hash and sparks handling of nulls when hashing sequences.
{code:java}
scala> spark.sql("SELECT hash('bar', null)").show()
+---------------+
|hash(bar, NULL)|
+---------------+
|    -1808790533|
+---------------+
scala> spark.sql("SELECT hash(null, 'bar')").show()
+---------------+
|hash(NULL, bar)|
+---------------+
|    -1808790533|
+---------------+
 {code}
these are differences sequences. e.g. these could be positions 0 and 1 in a dataframe which are diffferent columns with entirely different meanings. the hashes should not be the same.

another example:
{code:java}
scala> Seq(("john", null), (null, "john")).toDF("name", "alias").withColumn("hash", hash(col("name"), col("alias"))).show
+----+-----+---------+
|name|alias|     hash|
+----+-----+---------+
|john| null|487839701|
|null| john|487839701|
+----+-----+---------+ {code}
instead of ignoring nulls each null show do a transform to the hash so that the order of elements including the nulls matters for the outcome.
Bug: Avoid full outer join OOM on skewed dataset
Status: In Progress | Priority: Minor | Resolution: 
Created: 26/Jun/20 10:00
Affects Version/s: 2.3.1

SPARK-24985 changed {{SortMergeJoinExec}} to use {{ExternalAppendOnlyUnsafeRowArray}} on {{SortMergeFullOuterJoinScanner}}.

But its performance is very poor, because when matching the key, you need to constantly get the element at the specified position according to the index.


This PR add a {{FileBasedAppendOnlyUnsafeRowArray}}.

It can quickly locate the corresponding data according to the given index because of the offsets array in memory.

And in memory buffer rows can be limited by {{spark.sql.sortMergeJoinExec.buffer.in.memory.threshold}}.
Bug: Generate spark streaming test jar with maven plugin maven-jar-plugin
Status: Closed | Priority: Minor | Resolution: Cannot Reproduce
Created: 18/Jun/20 12:18
Affects Version/s: 3.0.0

When I compile streaming module alone, the test jar generated incorrect.
So we need to import maven plugin `maven-jar-plugin`
{code:java}

mvn clean package -DskipTests -Phive -Phive-thriftserver -Pyarn -DskipTests -pl streaming

[kun.wan@nta-offline-gw1 streaming]$ ll target/ -l
total 2668
drwxr-xr-x 2 kun.wan staff 4096 Jun 18 19:46 analysis
drwxr-xr-x 2 kun.wan staff 4096 Jun 18 19:46 antrun
drwxr-xr-x 2 kun.wan staff 4096 Jun 18 19:46 maven-archiver
drwxr-xr-x 3 kun.wan staff 4096 Jun 18 19:45 maven-shared-archive-resources
-rw-r--r-- 1 kun.wan staff 1134932 Jun 18 19:46 original-spark-streaming_2.12-3.0.0.jar
drwxr-xr-x 4 kun.wan staff 4096 Jun 18 19:46 scala-2.12
-rw-r--r-- 1 kun.wan staff 1137992 Jun 18 19:46 spark-streaming_2.12-3.0.0.jar
-rw-r--r-- 1 kun.wan staff 252535 Jun 18 19:46 spark-streaming_2.12-3.0.0-sources.jar
-rw-r--r-- 1 kun.wan staff 22 Jun 18 19:46 spark-streaming_2.12-3.0.0-tests.jar
-rw-r--r-- 1 kun.wan staff 170004 Jun 18 19:46 spark-streaming_2.12-3.0.0-test-sources.jar
drwxr-xr-x 2 kun.wan staff 4096 Jun 18 19:46 tmp
# After change maven plugin
[kun.wan@nta-offline-gw1 streaming]$ ll target/ -l
total 3500
drwxr-xr-x 2 kun.wan staff 4096 Jun 18 19:52 analysis
drwxr-xr-x 2 kun.wan staff 4096 Jun 18 19:51 antrun
drwxr-xr-x 2 kun.wan staff 4096 Jun 18 19:52 maven-archiver
drwxr-xr-x 3 kun.wan staff 4096 Jun 18 19:51 maven-shared-archive-resources
-rw-r--r-- 1 kun.wan staff 1134959 Jun 18 19:52 original-spark-streaming_2.12-3.0.0.jar
drwxr-xr-x 4 kun.wan staff 4096 Jun 18 19:51 scala-2.12
-rw-r--r-- 1 kun.wan staff 1138019 Jun 18 19:52 spark-streaming_2.12-3.0.0.jar
-rw-r--r-- 1 kun.wan staff 252558 Jun 18 19:52 spark-streaming_2.12-3.0.0-sources.jar
-rw-r--r-- 1 kun.wan staff 853239 Jun 18 19:52 spark-streaming_2.12-3.0.0-tests.jar
-rw-r--r-- 1 kun.wan staff 170027 Jun 18 19:52 spark-streaming_2.12-3.0.0-test-sources.jar
drwxr-xr-x 2 kun.wan staff 4096 Jun 18 19:51 tmp
{code}
Bug: GBTClassifier not working with OnevsRest
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 22/Jun/20 11:22
Affects Version/s: 2.4.0

I am trying to use GBT classifier for multi class classification using OnevsRest

 
{code:java}
from pyspark.ml.classification import MultilayerPerceptronClassifier,OneVsRest,GBTClassifier
from pyspark.ml import Pipeline,PipelineModel

lr = GBTClassifier(featuresCol='features', labelCol='label', predictionCol='prediction', maxDepth=5,                                                                                              maxBins=32,minInstancesPerNode=1, minInfoGain=0.0, maxMemoryInMB=256, cacheNodeIds=False,checkpointInterval=10, lossType='logistic', maxIter=20,stepSize=0.1, seed=None,subsamplingRate=1.0, featureSubsetStrategy='auto')

classifier = OneVsRest(featuresCol='features', labelCol='label', predictionCol='prediction', classifier=lr,    weightCol=None,parallelism=1)

pipeline = Pipeline(stages=[str_indxr,ohe,vecAssembler,normalizer,classifier])
model = pipeline.fit(train_data)
{code}
 

 

When I try this I get this error:

/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/spark/python/pyspark/ml/classification.py in _fit(self, dataset)
 1800 classifier = self.getClassifier()
 1801 assert isinstance(classifier, HasRawPredictionCol),\
 -> 1802 "Classifier %s doesn't extend from HasRawPredictionCol." % type(classifier)
 1803 
 1804 numClasses = int(dataset.agg(\{labelCol: "max"}).head()["max("+labelCol+")"]) + 1

AssertionError: Classifier <class 'pyspark.ml.classification.GBTClassifier'> doesn't extend from HasRawPredictionCol.
Bug: Replace decimal by int op in `make_interval` and `make_timestamp`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jun/20 12:43
Affects Version/s: 3.1.0

Currently, the MakeInterval and MakeTimestamp expressions use Decimal op in the conversion of seconds with fraction to microseconds. That's expensive op is not not needed and can be replaced by Int op.
Bug: Spark sequence() fails if start and end of range are identical dates
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jun/20 20:45
Affects Version/s: 2.4.0

 

The following Spark SQL query throws an exception
{code:java}
select sequence(cast("2011-03-01" as date), cast("2011-03-01" as date), interval 1 month)
{code}
The error is:

 

 
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 1java.lang.ArrayIndexOutOfBoundsException: 1 at scala.runtime.ScalaRunTime$.array_update(ScalaRunTime.scala:92) at org.apache.spark.sql.catalyst.expressions.Sequence$TemporalSequenceImpl.eval(collectionOperations.scala:2681) at org.apache.spark.sql.catalyst.expressions.Sequence.eval(collectionOperations.scala:2514) at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:389){noformat}
Bug: Specify the version of ChromeDriver and RemoteWebDriver which can work with guava 14.0.1
Status: In Progress | Priority: Minor | Resolution: 
Created: 16/Jun/20 04:43
Affects Version/s: 3.1.0

SPARK-31765 upgraded HtmlUnit due to a security reason and Selenium was also needed to be upgraded to work with the upgraded HtmlUnit.

After upgrading Selenium, ChromeDriver and RemoteWebDriver are implicitly upgraded because of dependency and the the implicitly upgraded modules can't work with guava 14.0.1 due to an API compatibility so we need to run ChromeUISeleniumSuite with a guava version specified like -Dguava.version=25.0-jre.
{code:java}
$ build/sbt -Dguava.version=25.0-jre -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest.default.exclude.tags= "testOnly org.apache.spark.ui.UISeleniumSuite"{code}
It's a little bit inconvenience so let's use older version which can work with guava 14.0.1.
Bug: Docker image should use `https` urls for only mirrors that support it(SSL)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jun/20 10:04
Affects Version/s: 3.0.0

It appears, that security.debian.org does not support https.
{code}
curl https://security.debian.org
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to security.debian.org:443 
{code}

While building the image, it fails in the following way.
{code}
MacBook-Pro:spark prashantsharma$ bin/docker-image-tool.sh -r scrapcodes -t v3.1.0-1 build
Sending build context to Docker daemon  222.1MB
Step 1/18 : ARG java_image_tag=8-jre-slim
Step 2/18 : FROM openjdk:${java_image_tag}
 ---> 381b20190cf7
Step 3/18 : ARG spark_uid=185
 ---> Using cache
 ---> 65c06f86753c
Step 4/18 : RUN set -ex &&     sed -i 's/http:/https:/g' /etc/apt/sources.list &&     apt-get update &&     ln -s /lib /lib64 &&     apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps &&     mkdir -p /opt/spark &&     mkdir -p /opt/spark/examples &&     mkdir -p /opt/spark/work-dir &&     touch /opt/spark/RELEASE &&     rm /bin/sh &&     ln -sv /bin/bash /bin/sh &&     echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su &&     chgrp root /etc/passwd && chmod ug+rw /etc/passwd &&     rm -rf /var/cache/apt/*
 ---> Running in a3461dadd6eb
+ sed -i s/http:/https:/g /etc/apt/sources.list
+ apt-get update
Ign:1 https://security.debian.org/debian-security buster/updates InRelease
Err:2 https://security.debian.org/debian-security buster/updates Release
  Could not handshake: The TLS connection was non-properly terminated. [IP: 151.101.0.204 443]
Get:3 https://deb.debian.org/debian buster InRelease [121 kB]
Get:4 https://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 https://deb.debian.org/debian buster/main amd64 Packages [7905 kB]
Get:6 https://deb.debian.org/debian buster-updates/main amd64 Packages [7868 B]
Reading package lists...
E: The repository 'https://security.debian.org/debian-security buster/updates Release' does not have a Release file.
The command '/bin/sh -c set -ex &&     sed -i 's/http:/https:/g' /etc/apt/sources.list &&     apt-get update &&     ln -s /lib /lib64 &&     apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps &&     mkdir -p /opt/spark &&     mkdir -p /opt/spark/examples &&     mkdir -p /opt/spark/work-dir &&     touch /opt/spark/RELEASE &&     rm /bin/sh &&     ln -sv /bin/bash /bin/sh &&     echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su &&     chgrp root /etc/passwd && chmod ug+rw /etc/passwd &&     rm -rf /var/cache/apt/*' returned a non-zero code: 100
Failed to build Spark JVM Docker image, please refer to Docker build output for details.
{code}

So, if we limit the https support to only deb.debian.org, that does the trick.
Bug: Summary.totalIterations greater than maxIters
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jun/20 07:05
Affects Version/s: 2.4.6

I am not sure whether it is a bug, but if we set *maxIter=n* in LiR/LiR/etc, the model.summary.totalIterations will return *n+1* if the training procedure does not drop out.

 

friendly ping [~huaxingao]
Bug: Add a class in spark with all Kafka configuration key available as string
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 10/Jun/20 20:41
Affects Version/s: 2.4.6

Add a class in spark with all Kafka configuration key available as string.

see the highligted class which i want.

eg:-

Current code:-

val df_cluster1 = spark
.read
.format("kafka")
.option("kafka.bootstrap.servers","cluster1_host:cluster1_port)
.option("subscribe", "topic1")

Expected code:-

val df_cluster1 = spark
.read
.format("kafka")
.option(*KafkaConstantClass*.KAFKA_BOOTSTRAP_SERVERS,"cluster1_host:cluster1_port)
.option(*KafkaConstantClass*.subscribe, "topic1")
Bug: Revert SPARK-31864 Adjust AQE skew join trigger condition
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jun/20 11:23
Affects Version/s: 3.0.0

As discussed in [https://github.com/apache/spark/pull/28669#issuecomment-641044531], revert SPARK-31864 for optimizing skew join to work for extremely clustered keys.
Bug: Handling the exception in SparkUI for getSparkUser method
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jun/20 09:46
Affects Version/s: 2.4.6

After SPARK-31632 SparkException is thrown from  def applicationInfo(

{code:java}
  def applicationInfo(): v1.ApplicationInfo = {
    try {
      // The ApplicationInfo may not be available when Spark is starting up.
      store.view(classOf[ApplicationInfoWrapper]).max(1).iterator().next().info
    } catch {
      case _: NoSuchElementException =>
        throw new SparkException("Failed to get the application information. " +
          "If you are starting up Spark, please wait a while until it's ready.")
    }
  }
{code}

Where as the caller for this method def getSparkUser in Spark UI is not handling SparkException in the catch


{code:java}
  def getSparkUser: String = {
    try {
      Option(store.applicationInfo().attempts.head.sparkUser)
        .orElse(store.environmentInfo().systemProperties.toMap.get("user.name"))
        .getOrElse("<unknown>")
    } catch {
      case _: NoSuchElementException => "<unknown>"
    }
  }
{code}

So On using this method (getSparkUser )we can get the application erred out.

So either we should thow


{code:java}
 throw new NoSuchElementException("Failed to get the application information. " +
          "If you are starting up Spark, please wait a while until it's ready.")
{code}

or else add the scenario to catch spark exception in getSparkUser
case _: SparkException => "<unknown>"
Bug: Fix the wrong coloring of nodes in DAG-viz
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jun/20 11:07
Affects Version/s: 3.0.0

In the Job Page and Stage Page, nodes which are associated with "barrier mode" in the DAG-viz will be colored pale green.

But, with some type of jobs, nodes which are not associated with the mode will also colored.

You can reproduce with the following operation.
{code:java}
sc.parallelize(1 to 10).barrier.mapPartitions(identity).repartition(1).collect() {code}
Bug: Cleanup locks and docs in CoarseGrainedSchedulerBackend
Status: Open | Priority: Trivial | Resolution: 
Created: 26/Jun/20 20:20
Affects Version/s: 3.1.0

Our locking is inconsistent inside of CoarseGrainedSchedulerBackend. In production this doesn't matter because the methods in are called from inside of ThreadSafeRPC but it would be better to cleanup the locks and docs to be consistent especially since other schedulers inherit from CoarseGrainedSchedulerBackend.
Bug: Simplify ArrowColumnVector ListArray accessor
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 24/Jun/20 02:54
Affects Version/s: 3.0.0

The ArrowColumnVector ListArray accessor calculates start and end offset indices manually. There were APIs added in Arrow 0.15.0 that do this and using them will simplify this code and make use of more stable APIs.
Bug: Spark SQL functions documentation refers to SQL API documentation without linking to it
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 04/Jun/20 20:04
Affects Version/s: 2.4.5

h2. Problem description

[The org.apache.spark.sql.functions documentation|http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions$] claims that:
{quote}(...) You can find the entire list of functions at SQL API documentation.

(...)

regr_count is an example of a function that is built-in but not defined here, because it is less commonly used. To invoke it, use expr("regr_count(yCol, xCol)").
{quote}
h2. Issue 1

I had to roam a little bit to arrive at: [https://spark.apache.org/docs/latest/api/sql/index.html]. Is this the "SQL API documentation" to which the text refers? If so, I suggest adding this link there.
h2. Issue 2

The documentation cites the function 'regr_count', but I couldn't find its documentation in the SQL API https://spark.apache.org/docs/latest/api/sql/index.html. The issue https://issues.apache.org/jira/browse/SPARK-23907 is related to this function. Where is the documentation for this function?
Bug: Broken link to java source code in Spark Scala api doc
Status: Open | Priority: Trivial | Resolution: 
Created: 26/Jun/20 06:48
Affects Version/s: 2.4.4

Java source code link in Spark api scaladoc is broken. This issue still persists for all Java files referred from scala doc in versions 2.4.4 & 2.4.5

The relative path expression "€\{FILE_PATH}.scala" in [https://github.com/apache/spark/blob/master/project/SparkBuild.scala] has ".scala" hardcoded in the end. 

If I try to access the source link on [https://spark.apache.org/docs/2.4.4/api/scala/index.html#org.apache.spark.sql.sources.v2.DataSourceOptions], it tries to take me to [https://github.com/apache/spark/tree/v2.4.4/sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceOptions.java.scala]

This is coming from /project/SparkBuild.scala :

Line # 720
{code:scala}
// Use GitHub repository for Scaladoc source links
unidocSourceBase := s"https://github.com/apache/spark/tree/v${version.value}",

scalacOptions in (ScalaUnidoc, unidoc) ++= Seq(
"-groups", // Group similar methods together based on the @group annotation.
"-skip-packages", "org.apache.hadoop"
) ++ (
// Add links to sources when generating Scaladoc for a non-snapshot release
if (!isSnapshot.value) {
Opts.doc.sourceUrl(unidocSourceBase.value + "€{FILE_PATH}.scala")
} else {
Seq()
}
){code}
Bug: [SS] 2.4 Kafka continuous processing print mislead initial offsets log 
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Jun/20 17:30
Affects Version/s: 2.4.6

When using structured streaming in continuous processing mode, after restart spark job, spark job can correctly pick up offsets in checkpoint location from last epoch. But it always print out below log:

20/06/12 00:58:09 INFO [stream execution thread for [id = 34e5b909-f9fe-422a-89c0-081251a68693, runId = 0246e19d-aaa1-4a5c-9091-bab1a0578a0a]] kafka010.KafkaContinuousReader: Initial offsets: \{"kafka_topic":{"8":51618236,"11":51610655,"2":51622889,"5":51637171,"14":51637346,"13":51627784,"4":51606960,"7":51632475,"1":51636129,"10":51632212,"9":51634107,"3":51611013,"12":51626567,"15":51640774,"6":51637823,"0":51629106}}

This log is misleading as spark didn't use this one as initial offsets. Also, it results in unnecessary kafka offset fetch. This is caused by below code in KafkaContinuousReader
{code:java}
offset = start.orElse {
  val offsets = initialOffsets match {
    case EarliestOffsetRangeLimit =>
      KafkaSourceOffset(offsetReader.fetchEarliestOffsets())
    case LatestOffsetRangeLimit =>
      KafkaSourceOffset(offsetReader.fetchLatestOffsets(None))
    case SpecificOffsetRangeLimit(p) =>
      offsetReader.fetchSpecificOffsets(p, reportDataLoss)
  }
  logInfo(s"Initial offsets: $offsets")
  offsets
}
{code}
 The code inside orElse block is always executed even when start has value.
Bug: Fix hyper links in SparkR documentation for CRAN submission
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 06/Jan/21 04:56
Affects Version/s: 3.1.0

CRAN submission fails due to:

{code}
   Found the following (possibly) invalid URLs:
     URL: http://jsonlines.org/ (moved to https://jsonlines.org/)
       From: man/read.json.Rd
             man/write.json.Rd
       Status: 200
       Message: OK
     URL: https://dl.acm.org/citation.cfm?id=1608614 (moved to
https://dl.acm.org/doi/10.1109/MC.2009.263)
       From: inst/doc/sparkr-vignettes.html
       Status: 200
       Message: OK
{code}