av9ash/GitBugs huggingface dataset:
Bug: Support data masking built-in function 'mask_last_n'
Status: Open | Priority: Minor | Resolution: 
Created: 07/Oct/22 00:15
Affects Version/s: 3.4.0

Support data masking built-in function *mask_last_n*

Return a masked version of str with the last n values masked. Upper case letters should be converted to "X", lower case letters should be converted to "x" and numbers should be converted to "n". For example, mask_last_n("1234-5678-8765-4321", 4) results in 1234-5678-8765-nnnn.
Bug: wrong doc on memory tuning regarding String object memory size, changed since version>=9
Status: Resolved | Priority: Trivial | Resolution: Won't Fix
Created: 03/Oct/22 20:24
Affects Version/s: 3.1.0

The documentation is wrong regarding memory consumption of java.lang.String
https://spark.apache.org/docs/latest/tuning.html#memory-tuning

internally, the source for this doc section is written here:
https://github.com/apache/spark/blob/master/docs/tuning.md?plain=1#L100

{noformat}
* Java `String`s have about 40 bytes of overhead over the raw string data (since they store it in an
  array of `Char`s and keep extra data such as the length), and store each character
  as *two* bytes due to `String`'s internal usage of UTF-16 encoding. Thus a 10-character string can
  easily consume 60 bytes.
{noformat}


reason: since java version >= 9 ... Java has optimized the problem described in the doc.
It used to be 16 bytes of header + using internally char coded as UTF-16

Notice  that before jdk 9 (since jdk 6, there was also an internal flag for HotSpot JVM :  -XX:+UseCompressedStrings , but it was not enabled by default ) 

Since OpenJdk >= 9...  with the implementation of JEP 254 ( https://openjdk.org/jeps/254 ), Strings are now internally encoded in UTF8 when they are simple Latin1 text, otherwise as before. There is now an extra byte   field in class java.lang.String to say if the "coder" is optimized for Latin1.

This field is described here in OpenJdk source code: https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/String.java#L170
The computation for the memory size of String was   "40+2*charCount" ... it is now "44+1*charCount" when it is Latin1 text, else "44+2*charCount" when it is not Latin1 text

the object overhead is 44 because of alignment... not 40+1 for adding one "byte" field
Bug: Clean up deprecated api usage in SparkThrowableSuite
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 19/Oct/22 06:05
Affects Version/s: 3.4.0

{code:java}
[WARNING] /Users/yangjie01/SourceCode/git/spark-mine/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala:77: [deprecation @ org.apache.spark.SparkThrowableSuite.<local SparkThrowableSuite>.errorClassFileContents | origin=org.apache.commons.io.IOUtils.toString | version=] method toString in class IOUtils is deprecated
[WARNING] /Users/yangjie01/SourceCode/git/spark-mine/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala:104: [deprecation @ org.apache.spark.SparkThrowableSuite.<local SparkThrowableSuite>.errorClassReadMeContents | origin=org.apache.commons.io.IOUtils.toString | version=] method toString in class IOUtils is deprecated
[WARNING] /Users/yangjie01/SourceCode/git/spark-mine/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala:325: [deprecation @ org.apache.spark.SparkThrowableSuite.<local SparkThrowableSuite>.$anonfun | origin=org.apache.commons.io.FileUtils.writeStringToFile | version=] method writeStringToFile in class FileUtils is deprecated
[WARNING] /Users/yangjie01/SourceCode/git/spark-mine/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala:335: [deprecation @ org.apache.spark.SparkThrowableSuite.<local SparkThrowableSuite>.$anonfun.reader | origin=java.io.File.toURL | version=] method toURL in class File is deprecated
[WARNING] /Users/yangjie01/SourceCode/git/spark-mine/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala:343: [deprecation @ org.apache.spark.SparkThrowableSuite.<local SparkThrowableSuite>.$anonfun | origin=org.apache.commons.io.FileUtils.writeStringToFile | version=] method writeStringToFile in class FileUtils is deprecated
[WARNING] /Users/yangjie01/SourceCode/git/spark-mine/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala:354: [deprecation @ org.apache.spark.SparkThrowableSuite.<local SparkThrowableSuite>.$anonfun.e | origin=java.io.File.toURL | version=] method toURL in class File is deprecated
[WARNING] /Users/yangjie01/SourceCode/git/spark-mine/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala:362: [deprecation @ org.apache.spark.SparkThrowableSuite.<local SparkThrowableSuite>.$anonfun | origin=org.apache.commons.io.FileUtils.writeStringToFile | version=] method writeStringToFile in class FileUtils is deprecated
[WARNING] /Users/yangjie01/SourceCode/git/spark-mine/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala:380: [deprecation @ org.apache.spark.SparkThrowableSuite.<local SparkThrowableSuite>.$anonfun.e | origin=java.io.File.toURL | version=] method toURL in class File is deprecated {code}
Bug: Estimated size in log message can overflow Int
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 12/Oct/22 14:52
Affects Version/s: 3.3.0

The log message from
```
             logInfo(s"Started reading broadcast variable $id with $numBlocks pieces " +
               s"(estimated total size $estimatedTotalSize)")

```
multiples two ints to calculate `estimatedTotalSize` which will overflow when the broadcast exceeds 2Gib.
Bug: to_timestamp failed to parse 2020-01-27T20:06:11.847-0800 using pattern "yyyy-MM-dd'T'HH:mm:ss.SSSz"
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 28/Jan/20 22:11
Affects Version/s: 3.0.0

{code:java}
SELECT to_timestamp("2020-01-27T20:06:11.847-0800", "yyyy-MM-dd'T'HH:mm:ss.SSSz")
{code}

This can return a valid value in Spark 2.4 but return NULL in the latest master

**2.4.5 RC2**
{code}
scala> sql("""SELECT to_timestamp("2020-01-27T20:06:11.847-0800", "yyyy-MM-dd'T'HH:mm:ss.SSSz")""").show
+----------------------------------------------------------------------------+
|to_timestamp('2020-01-27T20:06:11.847-0800', 'yyyy-MM-dd\'T\'HH:mm:ss.SSSz')|
+----------------------------------------------------------------------------+
|                                                         2020-01-27 20:06:11|
+----------------------------------------------------------------------------+
{code}

**2.2.3 ~ 2.4.4** (2.0.2 ~ 2.1.3 doesn't have `to_timestamp`).
{code}
spark-sql> SELECT to_timestamp("2020-01-27T20:06:11.847-0800", "yyyy-MM-dd'T'HH:mm:ss.SSSz");
2020-01-27 20:06:11
{code}
Bug: Streaming limit after streaming dropDuplicates can throw error
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 28/Jan/20 09:08
Affects Version/s: 2.3.0

{{LocalLimitExec}} does not consume the iterator of the child plan. So if there is a limit after a stateful operator like streaming dedup in append mode (e.g. {{streamingdf.dropDuplicates().limit(5}})), the state changes of streaming duplicate may not be committed (most stateful ops commit state changes only after the generated iterator is fully consumed). This leads to the next batch failing with {{java.lang.IllegalStateException: Error reading delta file .../N.delta does not exist}} as the state store delta file was never generated.
Bug: Limit after on streaming dataframe before streaming agg returns wrong results
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 28/Jan/20 09:12
Affects Version/s: 2.3.0

Limit before a streaming aggregate (i.e. {{df.limit(5).groupBy().count()}}) in complete mode was not being planned as a streaming limit. The planner rule planned a logical limit with a stateful streaming limit plan only if the query is in append mode. As a result, instead of allowing max 5 rows across batches, the planned streaming query was allowing 5 rows in every batch thus producing incorrect results.
Bug: Wrong result of the combination of from_utc_timestamp and to_utc_timestamp
Status: In Progress | Priority: Major | Resolution: 
Created: 31/Jan/20 21:10
Affects Version/s: 2.4.4

Applying to_utc_timestamp() to results of from_utc_timestamp() should return the original timestamp in the same time zone. In the range of 100 years, the combination of functions returns wrong results 280 times out of 1753200:
{code:java}
scala> val SECS_PER_YEAR = (36525L * 24 * 60 * 60)/100
SECS_PER_YEAR: Long = 31557600

scala> val SECS_PER_MINUTE = 60L
SECS_PER_MINUTE: Long = 60

scala>  val tz = "America/Los_Angeles"
tz: String = America/Los_Angeles

scala> val df = spark.range(-50 * SECS_PER_YEAR, 50 * SECS_PER_YEAR, 30 * SECS_PER_MINUTE)
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> val diff = df.select((to_utc_timestamp(from_utc_timestamp($"id".cast("timestamp"), tz), tz).cast("long") - $"id").as("diff")).filter($"diff" !== 0)
warning: there was one deprecation warning; re-run with -deprecation for details
diff: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [diff: bigint]

scala> diff.count
res14: Long = 280

scala> df.count
res15: Long = 1753200
{code}
Bug: SPIP: Support push-based shuffle to improve shuffle efficiency
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/20 01:15
Affects Version/s: 3.1.0

In a large deployment of a Spark compute infrastructure, Spark shuffle is becoming a potential scaling bottleneck and a source of inefficiency in the cluster. When doing Spark on YARN for a large-scale deployment, people usually enable Spark external shuffle service and store the intermediate shuffle files on HDD. Because the number of blocks generated for a particular shuffle grows quadratically compared to the size of shuffled data (# mappers and reducers grows linearly with the size of shuffled data, but # blocks is # mappers * # reducers), one general trend we have observed is that the more data a Spark application processes, the smaller the block size becomes. In a few production clusters we have seen, the average shuffle block size is only 10s of KBs. Because of the inefficiency of performing random reads on HDD for small amount of data, the overall efficiency of the Spark external shuffle services serving the shuffle blocks degrades as we see an increasing # of Spark applications processing an increasing amount of data. In addition, because Spark external shuffle service is a shared service in a multi-tenancy cluster, the inefficiency with one Spark application could propagate to other applications as well.

In this ticket, we propose a solution to improve Spark shuffle efficiency in above mentioned environments with push-based shuffle. With push-based shuffle, shuffle is performed at the end of mappers and blocks get pre-merged and move towards reducers. In our prototype implementation, we have seen significant efficiency improvements when performing large shuffles. We take a Spark-native approach to achieve this, i.e., extending Spark’s existing shuffle netty protocol, and the behaviors of Spark mappers, reducers and drivers. This way, we can bring the benefits of more efficient shuffle in Spark without incurring the dependency or overhead of either specialized storage layer or external infrastructure pieces.

 

Link to dev mailing list discussion: [http://apache-spark-developers-list.1001551.n3.nabble.com/Enabling-push-based-shuffle-in-Spark-td28732.html]
Bug: remove dependency on jackson-mapper-asl-1.9.13 and jackson-core-asl-1.9.13
Status: Open | Priority: Major | Resolution: 
Created: 09/Jan/20 02:40
Affects Version/s: 2.4.4

These 2 libraries are deprecated and replaced by the jackson-databind libraries which are already included.  These two libraries are flagged by our vulnerability scanners as having the following security vulnerabilities.  I've set the priority to Major due to the Critical nature and hopefully they can be addressed quickly.  Please note, I'm not a developer but work in InfoSec and this was flagged when we incorporated spark into our product.  If you feel the priority is not set correctly please change accordingly.  I'll watch the issue and flag our dev team to update once resolved.  

jackson-mapper-asl-1.9.13

CVE-2018-7489 (CVSS 3.0 Score 9.8 CRITICAL)

[https://nvd.nist.gov/vuln/detail/CVE-2018-7489] 

 

CVE-2017-7525 (CVSS 3.0 Score 9.8 CRITICAL)

[https://nvd.nist.gov/vuln/detail/CVE-2017-7525]

 

CVE-2017-17485 (CVSS 3.0 Score 9.8 CRITICAL)

[https://nvd.nist.gov/vuln/detail/CVE-2017-17485]

 

CVE-2017-15095 (CVSS 3.0 Score 9.8 CRITICAL)

[https://nvd.nist.gov/vuln/detail/CVE-2017-15095]

 

CVE-2018-5968 (CVSS 3.0 Score 8.1 High)

[https://nvd.nist.gov/vuln/detail/CVE-2018-5968]

 

jackson-core-asl-1.9.13

CVE-2016-7051 (CVSS 3.0 Score 8.6 High)

https://nvd.nist.gov/vuln/detail/CVE-2016-7051
Bug: Two Spark structured streaming jobs cannot write to same base path
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Jan/20 07:44
Affects Version/s: 2.3.0

Hi All,

Spark Structured Streaming doesn't allow two structured streaming jobs to write data to the same base directory which is possible with using dstreams.

As __spark___metadata directory will be created by default for one job, second job cannot use the same directory as base path as already _spark__metadata directory is created by other job, It is throwing exception.

Is there any workaround for this, other than creating separate base path's for both the jobs.

Is it possible to create the __spark__metadata directory else where or disable without any data loss.

If I had to change the base path for both the jobs, then my whole framework will get impacted, So i don't want to do that.
Bug: PySpark enum subclass crashes when used inside UDF
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 09/Jan/20 16:52
Affects Version/s: 2.4.4

PySpark enum subclass crashes when used inside a UDF.

 

Example:
{code:java}
from enum import Enum
class Direction(Enum):
    NORTH = 0
    SOUTH = 1
{code}
 

Working:
{code:java}
Direction.NORTH{code}
 

Crashing:
{code:java}
@udf
def fn(a):
    Direction.NORTH
    return ""

df.withColumn("test", fn("a")){code}
 

Stacktrace:
{noformat}
SparkException: Job aborted due to stage failure: Task 0 in stage 9.0 failed 4 times, most recent failure: Lost task 0.3 in stage 9.0 (TID 235, 10.139.64.21, executor 0): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
    File "/databricks/spark/python/pyspark/serializers.py", line 182, in _read_with_length return self.loads(obj)
    File "/databricks/spark/python/pyspark/serializers.py", line 695, in loads return pickle.loads(obj, encoding=encoding)
    File "/databricks/python/lib/python3.7/enum.py", line 152, in __new__ enum_members = {k: classdict[k] for k in classdict._member_names}
AttributeError: 'dict' object has no attribute '_member_names'{noformat}
 

I suspect the problem is in *python/pyspark/cloudpickle.py*. On line 586 in the function *_save_dynamic_enum*, the attribute *_member_names* is removed from the enum. Yet, this attribute is required by the *Enum* class. This results in all Enum subclasses crashing.
Bug: Question about spark on k8s
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 15/Jan/20 01:33
Affects Version/s: 3.0.0

My question is, why we wrote the domain name of Kube-DNS in the code? Isn't
it better to read domain name from the service, or just use the hostname?

In our scenario, we run spark on Kata-like containers, and found the code
had written the Kube-DNS domain. If Kube-DNS is not configured in
environment, tasks would run failed.

Besides, kube-dns is just a plugin for k8s, not a required component for k8s. We can use better DNS services without necessarily using this.
Bug: Random pyspark-shell applications being generated
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 17/Jan/20 10:12
Affects Version/s: 2.3.2

 

When we submit a particular spark job, this happens. Not sure from where these pyspark-shell applications get generated, but they persist for like 5s and gets killed. We're not able to figure put why this happens
Bug: StorageLevel.DISK_ONLY_2 causes the data loss
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 20/Jan/20 03:45
Affects Version/s: 2.3.0

As shown in the attachment，after I load the data of the hive table which is immutable and cache the data in the level of the DISK_ONLY_2，the count value of the data  is different every time.
Bug: start spark applications without a 30second startup penalty
Status: Open | Priority: Major | Resolution: 
Created: 18/Jan/20 12:22
Affects Version/s: 3.1.0

see https://stackoverflow.com/questions/57610138/how-to-start-spark-applications-without-a-30second-startup-penalty

using spark standalone.

There are several sleeps that can be removed:

grep -i 'sleep(' -R * | grep -v 'src/test/' | grep -E '^core' | grep -ivE 'mesos|yarn|python|HistoryServer|spark/ui/'
core/src/main/scala/org/apache/spark/util/Clock.scala:      Thread.sleep(sleepTime)
core/src/main/scala/org/apache/spark/SparkContext.scala:   * sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count()
core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala:  private def delay(secs: Duration = 5.seconds) = Thread.sleep(secs.toMillis)
core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala:          Thread.sleep(1000)
core/src/main/scala/org/apache/spark/deploy/Client.scala:    Thread.sleep(5000)
core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:      Thread.sleep(100)
core/src/main/scala/org/apache/spark/deploy/StandaloneResourceUtils.scala:          Thread.sleep(duration)
core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala:    def sleep(seconds: Int): Unit = (0 until seconds).takeWhile { _ =>
core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala:      Thread.sleep(1000)
core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala:        sleeper.sleep(waitSeconds)
core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala:  def sleep(seconds: Int): Unit
core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:      Thread.sleep(REPORT_DRIVER_STATUS_INTERVAL)
core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala:      Thread.sleep(10)
core/src/main/scala/org/apache/spark/storage/BlockManager.scala:          Thread.sleep(SLEEP_TIME_SECS * 1000L)
Bug: Postgres Column Interval converts to string and cant be written back to postgres
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 22/Jan/20 20:25
Affects Version/s: 2.4.4

If we read a  "Interval" type column from postgres and try to save it back to postgres, an exception is occured as during read operation postgres column is converted to String and while saving back it gives an error

 

java.sql.BatchUpdateException: Batch entry 0 INSERT INTO test_table ("dob","dob_time","dob_time_zone","duration") VALUES ('2019-05-29 -04','2016-08-12 10:22:31.100000-04','2016-08-12 13:22:31.100000-04','3 days 10:00:00') was aborted: ERROR: column "duration" is of type interval but expression is of type character varying
 Hint: You will need to rewrite or cast the expression.
 Position: 86 Call getNextException to see other errors in the batch.
 at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:151)
 at org.postgresql.core.ResultHandlerDelegate.handleError(ResultHandlerDelegate.java:45)
 at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2159)
 at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:463)
 at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:794)
 at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1662)
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:672)
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:834)
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:834)
 at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
 at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
 at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
 at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
 at org.apache.spark.scheduler.Task.run(Task.scala:123)
 at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
 at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
Bug: Add a Google Maven Central as a primary repository
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/20 01:09
Affects Version/s: 2.4.5

See [http://apache-spark-developers-list.1001551.n3.nabble.com/Adding-Maven-Central-mirror-from-Google-to-the-build-td28728.html]

This Jira targets to switch the main repo to Google Maven Central.
Bug: The same job will be computated for many times when using Dataset.show()
Status: Open | Priority: Major | Resolution: 
Created: 07/Jan/20 01:55
Affects Version/s: 3.1.0

When I run the example sql.SparkSQLExample, df.show() at line 60 would trigger an action. On WebUI, I noticed that this API creates 5 jobs, all of which have the same lineage graph with the same RDDs and the same call stacks. That means Spark recomputates the job for 5 times. But strangely, sqlDF.show() at line 123 only creates 1 job.

I don't know what happened at show() at line 60.
Bug: Write mode ignored when using CodecStreams
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 06/Jan/20 20:29
Affects Version/s: 2.4.4

Overwrite is hardcoded to false in the codec stream. This can cause issues, particularly with aws tools, that make it impossible to retry.

Ideally, this should be read from the write mode set for the DataWriter that is writing through this codec class.

[https://github.com/apache/spark/blame/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CodecStreams.scala#L81]
Bug: Use mvn in `dev/scalastyle`
Status: Closed | Priority: Major | Resolution: Invalid
Created: 16/Jan/20 23:04
Affects Version/s: 2.4.4

This issue aims to use `mvn` instead of `sbt`.

As of now, Apache Spark sbt build is broken by the Maven Central repository policy.

- https://stackoverflow.com/questions/59764749/requests-to-http-repo1-maven-org-maven2-return-a-501-https-required-status-an

> Effective January 15, 2020, The Central Maven Repository no longer supports insecure
> communication over plain HTTP and requires that all requests to the repository are 
> encrypted over HTTPS.

We can reproduce this locally by the following.

    $ rm -rf ~/.m2/repository/org/apache/apache/18/
    $ build/sbt clean

This issue aims to recover GitHub Action `lint-scala` first by using mvn.
Bug: org.slf4j.Logger and org.apache.commons.collections classes not built as part of hadoop-provided profile
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 23/Jan/20 10:20
Affects Version/s: 2.4.2

We're using spark-2.4.4-bin-without-hadoop.tgz and executing Java Word count (org.apache.spark.examples.JavaWordCount) example on local files.

But we're seeing that it is expecting org.slf4j.Logger and org.apache.commons.collections classes to be available for executing this.

We expected the binary to work as it is for local files. Is there anything which we're missing?