av9ash/GitBugs huggingface dataset:
Bug: ABFS: Add changes for expect hundred continue header with append requests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/22 05:56
Affects Version/s: 3.3.5

 Heavy load from a Hadoop cluster lead to high resource utilization at FE nodes. Investigations from the server side indicate payload buffering at Http.Sys as the cause. Payload of requests that eventually fail due to throttling limits are also getting buffered, as its triggered before FE could start request processing.

Approach: Client sends Append Http request with Expect header, but holds back on payload transmission until server replies back with HTTP 100. We add this header for all append requests so as to reduce.

We made several workload runs with and without hundred continue enabled and the overall observation is that :-
 # The ratio of TCP SYN packet count with and without expect hundred continue enabled is 0.32 : 3 on average.
 #  The ingress into the machine at TCP level is almost 3 times lesser with hundred continue enabled which implies a lot of bandwidth save.
Bug: Basic verification for the release candidate vote
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/22 06:56
Affects Version/s: 3.4.0

We should provide script for the basic sanity of Hadoop release candidates. It should include:
 * Signature
 * Checksum
 * Rat check
 * Build from src
 * Build tarball from src

 

Although we can include unit test as well, but overall unit test run is going to be significantly higher, and precommit Jenkins builds provide better view of UT sanity.
Bug: ViewFileSystem: Add Support for Localized Trash Root
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/22 19:08
Affects Version/s: 3.4.0

getTrashRoot() in ViewFileSystem calls getTrashRoot() from underlying filesystem, to return the trash root. Most of the time, we get a trash root in user home dir. This can lead to problems when an application wants to delete a file in a mounted point using moveToTrash() in TrashPolicyDefault, because we can not rename across multiple filesystems/hdfs namenodes. 

 

We propose the following extension to getTrashRoot/getTrashRoots in ViewFileSystem: add a flag to return a localized trash root for ViewFileSystem. A localized trash root is a trash root which starts from the root of a mount point (e.g., /mountpointRoot/.Trash/\{user}). 

* If CONFIG_VIEWFS_MOUNT_POINT_LOCAL_TRASH is not set to true, or
* when the path p is in a snapshot or an encryption zone, return
* the default trash root in user home dir.
*
* when CONFIG_VIEWFS_MOUNT_POINT_LOCAL_TRASH is set to true,
* 1) if path p is mounted from the same targetFS as user home dir,
* return a trash root in user home dir.
* 2) else, return a trash root in the mounted targetFS
*
Bug: Upgrade maven enforcer plugin and relevant dependencies
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/22 18:59
Affects Version/s: 3.4.0

Maven enforcer plugin's latest version 3.0.0 has some noticeable improvements (e.g. MENFORCER-350, MENFORCER-388, MENFORCER-353) and fixes for us to incorporate. Besides, some of the relevant enforcer dependencies (e.g. extra enforcer rules and restrict import enforcer) too have good improvements.

We should upgrade maven enforcer plugin and the relevant dependencies.
Bug: Update junit 5 version due to build issues
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Feb/22 13:12

{code:java}
Feb 11, 2022 11:31:43 AM org.junit.platform.launcher.core.DefaultLauncher handleThrowable WARNING: TestEngine with ID 'junit-vintage' failed to discover tests org.junit.platform.commons.JUnitException: Failed to parse version of junit:junit: 4.13.2 at org.junit.vintage.engine.JUnit4VersionCheck.parseVersion(JUnit4VersionCheck.java:54) {code}
[https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3980/1/artifact/out/patch-unit-root.txt]

seems like junit.vintage.version=5.5.1 is incompatible with junit.version=4.13.2

see 2nd answer on [https://stackoverflow.com/questions/59900637/error-testengine-with-id-junit-vintage-failed-to-discover-tests-with-spring]

my plan is to upgrade junit.vintage.version and junit.jupiter.version to 5.8.2
Bug: Boost S3A Stream Read Performance
Status: Open | Priority: Major | Resolution: 
Created: 28/Mar/22 11:28
Affects Version/s: 3.3.2

calibrate S3A input stream performance against recent applications/data formats and improve where necessary.

HADOOP-18028 is a key part of this, but there are other issues/opertunities

# we could add machine parsable trace-level logging in FSDataInputStream to collect stats on how stream apis are invoked, so collect data from real apps; analyze
# implement those APIs which some apps use (ByteBufferPositionedReadable), not so much for direct implementation as to get better information from the app as its read plan
# the `normal` mode doesn't switch from sequential on forward seeks. Is that always appropriate?
# choose different buffering options when doing whole file IO vs sequential vs random
Bug: Netty version 3.10.6 to be upgraded to handle CVE-2021-43797
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 11/Feb/22 08:03
Affects Version/s: 3.1.1

Netty version 3.10.6 to be upgraded to handle CVE-2021-43797, even Netty-4 upgraded to handle this issue.

Please feedback
Bug: Upgrade Netty to 4.1.77.Final
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/22 07:37
Affects Version/s: 3.3.3

h4. Netty version - 4.1.71 has fix some CVEs.

CVE-2019-20444,
CVE-2019-20445
CVE-2022-24823

Upgrade to latest version.
Bug: json smart 1.3.2 still appears in Trivy scan of build 
Status: Open | Priority: Major | Resolution: 
Created: 28/Feb/22 15:42
Affects Version/s: 3.3.2

when building 3.3.2 Hadoop is still failing CVE scans showing the following error. We are unable to use Hadoop with this CVE showing

 
"VulnerabilityID": "CVE-2021-31684", 
"PkgName": "net.minidev:json-smart", 
"PkgPath": "...lib/org.apache.hadoop.hadoop-client-runtime-3.3.2.jar", 
"InstalledVersion": "1.3.2", 
"FixedVersion": "2.4.5, 1.3.3",

 

more specifically

 
"VulnerabilityID": "CVE-2021-31684", 
"PkgName": "net.minidev:json-smart", 
"PkgPath": ".../lib/com.nimbusds.nimbus-jose-jwt-9.8.1.jar", 
"InstalledVersion": "1.3.2", 
"FixedVersion": "2.4.5, 1.3.3",
Bug: Handle memory fragmentation in S3 Vectored IO implementation.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/22 14:02

As we have implemented merging of ranges in the S3AInputStream implementation of vectored IO api, it can lead to memory fragmentation. Let me explain by example.

 

Suppose client requests for 3 ranges. 

0-500, 700-1000 and 1200-1500.

Now because of merging, all the above ranges will get merged into one and we will allocate a big byte buffer of 0-1500 size but return sliced byte buffers for the desired ranges.

Now once the client is done reading all the ranges, it will only be able to free the memory for requested ranges and memory of the gaps will never be released for eg here (500-700 and 1000-1200).
Bug: tune logging of prefetch problems
Status: Open | Priority: Major | Resolution: 
Created: 29/Mar/22 12:31
Affects Version/s: 3.4.0

we need to think what we want to do here.

* we would want errors to be counted and included in stream iostats
* but we've seen abfs apps flooded with stack traces when a transient network error breaks every prefetch.
* logging through LogExactlyOnce would highlight a problem, but on a long lived process, not show enough, unless the cache manager had a per instance value.

# PrefetchTask needs to be non static and update the CachingBlockManager when there is a problem; it can LogExactlyOnce
# stream statistics need to include cache values (hits, misses, prefech failures, how long futures were queued before execution...)
Bug: Verify FileUtils.unTar() handling of missing .tar files: Fixes CVE-2022-25168
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/22 17:02
Affects Version/s: 2.10.1

add a test to verify FileUtils.unTar() of a non .gz fails meaningfully if file isn't present; fix if not.

test both the unix and windows paths.

This patch contains the fix (and tests to verify it) for CVE-2022-25168

[mitre CVE|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-25168]

 hadoop branches without YARN-2185 are at risk in yarn downloads; those with the patch in are not


h2. Announcement
{code}
Severity: important

Versions affected:

2.0.0 to 2.10.1, 3.0.0-alpha to 3.2.3, 3.3.0 to 3.3.2

Description:

Apache Hadoop's FileUtil.unTar(File, File) API does not escape the
input file name before being passed to the shell. An attacker can
inject arbitrary commands.

This is only used in Hadoop 3.3
InMemoryAliasMap.completeBootstrapTransfer, which is only ever run by
a local user.

It has been used in Hadoop 2.x for yarn localization, which does
enable remote code execution.

It is used in Apache Spark, from the SQL command ADD ARCHIVE. As the
ADD ARCHIVE command adds new binaries to the classpath, being able to
execute shell scripts does not confer new permissions to the caller.

SPARK-38305. "Check existence of file before untarring/zipping", which
is included in 3.3.0, 3.1.4, 3.2.2, prevents shell commands being
executed, regardless of which version of the hadoop libraries are in
use.


Mitigation:

Users should upgrade to Apache Hadoop 2.10.2, 3.2.4, 3.3.3 or upper
(including HADOOP-18136).

Credit:

Apache Hadoop would like to thank Kostya Kortchinsky for reporting this issue

{code}
Bug: S3A auditing leaks memory through ThreadLocal references
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/22 15:34
Affects Version/s: 3.3.2

{{ActiveAuditManagerS3A}} uses thread locals to map to active audit spans, which (because they are wrapped) include back reference to the audit manager instance and the config it was created with.

these *do not* get cleaned up when the FS instance is closed.

if you have a long lived process creating and destroying many FS instances, then memory gets used up.

This fix moves off threadlocal into a map of weak references. while a strong reference is held` (for example in the s3a entry point method) then the references will always resolve. but if those are released then when a GC is triggered these weak references will not be retained, so not use up memory other than entries in the the ha!sh map. the map is held by the s3a auditing integration, so when the fs is closed, everything is freed up.

h3. backport/cherrypicking

if this is backported, followup with HADOOP-18456
Bug: S3 SDK Upgrade causes AccessPoint ARN endpoint mistranslation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/22 16:41
Affects Version/s: 3.3.5

Since upgrading the [SDK to 1.12.132|https://github.com/apache/hadoop/pull/3864] the access point endpoint translation was broken.

Correct endpoints should start with "s3-accesspoint.", after SDK upgrade they start with "s3.accesspoint-" which messes up tests + region detection by the SDK.
Bug: TemporaryAWSCredentialsProvider has no credentials
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 11/Jan/22 18:10
Affects Version/s: 3.3.1

Not quite sure how to phrase this bugreport but I'll give it a try..
We are using a SparkSession to access parquet files on AWS/S3

it is ok, if there is only one  s3a URL supplied
it used to be ok if there is a bunch of s3a URLs - that's broken siince hadoop:3.3.1

 

 

I've attached a sample script - yet it relys on spark+hadoop installed
Bug: CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client  
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 05/Jan/22 19:47
Affects Version/s: 3.3.1

Our static vulnerability scanner (Fortify On Demand) detected [NVD - CVE-2021-0341 (nist.gov)|https://nvd.nist.gov/vuln/detail/CVE-2021-0341#VulnChangeHistorySection] in our application. We traced the vulnerability to a transitive dependency coming from hadoop-hdfs-client, which depends on okhttp@2.7.5 ([hadoop/pom.xml at trunk · apache/hadoop (github.com)|https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L137]). To resolve this issue, okhttp should be upgraded to 4.9.2+ (ref: [CVE-2021-0341 · Issue #6724 · square/okhttp (github.com)|https://github.com/square/okhttp/issues/6724]).
Bug: Exclude com/jcraft/jsch classes from being shaded/relocated
Status: Open | Priority: Major | Resolution: 
Created: 22/Jan/22 09:17
Affects Version/s: 3.3.1

Spark 3.2.0 transitively introduces hadoop-client-api and hadoop-client-runtime dependencies.

When we create a SFTPFileSystem instance (org.apache.hadoop.fs.sftp.SFTPFileSystem) it tries to load the relocated classes from _com.jcraft.jsch_ package.

The filesystem instance creation fails with error:
{code:java}
java.lang.ClassNotFoundException: org.apache.hadoop.shaded.com.jcraft.jsch.SftpException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) {code}

Excluding client from transitive load of spark and directly using hadoop-common/hadoop-client is the way its working for us.
Bug: S3File to store reference to active S3Object in a field.
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 28/Mar/22 16:06
Affects Version/s: 3.4.0

HADOOP-17338 showed us how recent {{S3Object.finalize()}} can call stream.close() and so close an active stream if a GC happens during a read. replicate the same fix here.
Bug: Partial/Incomplete groups list can be returned in LDAP groups lookup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/22 10:30

Hello,

The  
{code:java}
Set<String> doGetGroups(String user, int goUpHierarchy) {code}
method in

[https://github.com/apache/hadoop/blob/b27732c69b114f24358992a5a4d170bc94e2ceaf/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java#L476]

Looks like having an issue if in the middle of the loop a *NamingException* is caught:

The groups variable is not reset in the catch clause and therefore the fallback lookup cannot be executed (when goUpHierarchy==0 at least):
||
{code:java}
if (groups.isEmpty() || goUpHierarchy > 0) {        
    groups = lookupGroup(result, c, goUpHierarchy);
}
{code}
 

Consequence is that only a partial list of groups is returned, which is not correct.

Following options could be used as solution:
 * Reset the group to an empty list in the catch clause, to trigger the fallback query.
 * Add an option flag to enable ignoring groups with Naming Exception (since they are not groups most probably)

Independently, would any issue also occur (and therefore full list cannot be returned) in the first lookup as well as in the fallback query, the method should/could(with option flag) throw an Exception, because in some scenario accuracy is important.
Bug: hadoop-yarn-ui has a number of insecure dependencies
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/22 01:00

Many of these are rates as critical or high risk vulnerabilities. This list is the tip of the iceberg.

Examples found by dependabot
* https://github.com/advisories/GHSA-35jh-r3h4-6jhm (lodash-es)
* https://github.com/advisories/GHSA-p6mc-m468-83gw (lodash)
* https://github.com/advisories/GHSA-pc58-wgmc-hfjr (mout)
* https://github.com/advisories/GHSA-4rq4-32rv-6wp6 (shelljs)
* https://github.com/advisories/GHSA-5955-9wpr-37jh (tar)
* https://github.com/advisories/GHSA-765h-qjxv-5f44 (handlebars)
* https://github.com/advisories/GHSA-f2jv-r9rf-7988 (handlebars)
* https://github.com/advisories/GHSA-xfhh-g9f5-x4m4 (socket.io-parser)
* https://github.com/advisories/GHSA-72mh-269x-7mh5 (xmlhttprequest-ssl)
* https://github.com/advisories/GHSA-g78m-2chm-r7qv (websocket-extensions)

may need to upgrade ember to allow some of these NPMs to be updated
Bug: `org.wildfly.openssl` should not be shaded by Hadoop build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/22 10:50
Affects Version/s: 3.3.1

`org.wildfly.openssl` is a runtime library and its references are being shaded on Hadoop, breaking the integration with other frameworks like Spark, whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error produced in this situation is:
{code:java}
Suppressed: java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code}
Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`. Spark tries to add it to its classpath without the shade, thus creating this issue.

Dependencies which are not on "compile" scope should probably not be shaded to avoid this kind of integration issues.
Bug: Fix failure of create-release script due to releasedocmaker changes in branch-2.10
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/22 05:31
Affects Version/s: 2.10.1

The file name generated by releasedocmaker of Yetus was changed from CHANGES.md to CHANGELOG.md. dev-support scripts should be updated along with the change.
Bug: fix bugs when looking up record from upstream DNS servers.
Status: Open | Priority: Major | Resolution: 
Created: 21/Jan/22 01:16
Affects Version/s: 3.1.2

When query A record which is chained by CNAME, YARN Registry DNS Server does not properly respond. Some CNAME records are missing.

For example, "repo.maven.apache.org" is chaned as follows:

repo.maven.apache.org.	21317	IN	CNAME	repo.apache.maven.org.
repo.apache.maven.org.	20114	IN	CNAME	maven.map.fastly.net.
maven.map.fastly.net.	7	IN	A	199.232.192.215
maven.map.fastly.net.	7	IN	A	199.232.196.215

If ask A record for "repo.maven.apache.org" using "dig" or "nslookup", YARN Registry DNS Server will give answers similar to this:
(10.1.2.3, 10.8.8.8 IP is virtual)


{code}
$ nslookup repo.maven.apache.org 10.1.2.3
Server:		10.1.2.3
Address:	10.1.2.3#53

Non-authoritative answer:
repo.maven.apache.org	canonical name = repo.apache.maven.org.
Name:	maven.map.fastly.net
Address: 151.101.196.215
** server can't find repo.apache.maven.org: NXDOMAIN
{code}

The reason why you can see "NXDOMAIN", "nslookup" will query "A" & "AAAA" records.
If there is no answer from other dns server, "answers == null" but YARN Registry DNS Server has a bug. There is no null handling.


{code:java}
    // Forward lookup to primary DNS servers
    Record[] answers = getRecords(name, type);
    try {
      for (Record r : answers) {
        if (!response.findRecord(r)) {
          if (r.getType() == Type.SOA) {
            response.addRecord(r, Section.AUTHORITY);
          } else {
            response.addRecord(r, Section.ANSWER);
          }
        }
        if (r.getType() == Type.CNAME) {
          Name cname = r.getName();
          if (iterations < 6) {
            remoteLookup(response, cname, type, iterations + 1);
          }
        }
      }
    } catch (NullPointerException e) {
      return Rcode.NXDOMAIN;
    } catch (Throwable e) {
      return Rcode.SERVFAIL;
    }
    return Rcode.NOERROR;
{code}





It should be like this:

{code}
nslookup repo.maven.apache.org 10.8.8.8
Server:		10.8.8.8
Address:	10.8.8.8#53

Non-authoritative answer:
repo.maven.apache.org	canonical name = repo.apache.maven.org.
repo.apache.maven.org	canonical name = maven.map.fastly.net.
Name:	maven.map.fastly.net
Address: 151.101.196.215
{code}

I will make a pull request at https://github.com/apache/hadoop soon.
Bug: document use and architecture design of prefetching s3a input stream
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/22 20:41
Affects Version/s: 3.4.0

Document S3PrefetchingInputStream for users  (including any new failure modes in troubleshooting) and the architecture for maintainers

there's some markdown in hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/README.md already
Bug: Fileutil's unzip method causes unzipped files to lose their original permissions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/22 07:41
Affects Version/s: 2.7.2

When Spark decompresses the zip file, if the original file has the executable permission, but the unzip method of FileUtil is invoked, the decompressed file loses the executable permission, we should save the original permission
Bug: s3a prefetching stream to move off twitter FuturePool
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 25/Mar/22 20:36
Affects Version/s: 3.4.0

This has to be a blocker for the merge I'm afraid: move off twitter's util lib and its future pool

it's not just another jar, its a full scala runtime. and as we know, that's a very brittle runtime. for existing scala code. that's their problem...we don't want to get involved in this

{code}
[INFO] +- com.twitter:util-core_2.11:jar:21.2.0:compile
[INFO] |  +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO] |  +- com.twitter:util-function_2.11:jar:21.2.0:compile
[INFO] |  +- org.scala-lang.modules:scala-collection-compat_2.11:jar:2.1.2:compile
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.12:compile
[INFO] |  \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.1.2:compile

{code}
Bug: hadoop-common enhancements for the Manifest Committer of MAPREDUCE-7341
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/22 14:47
Affects Version/s: 3.3.5

Make the necessary changes to hadoop-common to support the manifest committer of 
MAPREDUCE-7341j

* new stats names in StoreStatisticNames (for joint use with s3a committers)
* improvements to IOStatistics
* s3a committer task pool to move over as ThreadPool

this does not break s3a committer as they will only adopt these changes in HADOOP-17833
Bug: Fix ITestAuditManagerDisabled after S3A audit logging was enabled in HADOOP-18091
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/22 07:01
Affects Version/s: 3.3.5

In HADOOP-18094, we disabled audit logging by default and introduced an IT which verifies that by default the audit logging is disabled. After HADOOP-18091, we switched the audit logging back on, but that test remained, which now fails since by default we have the audit logging enabled now.

*StackTrace:*
{noformat}
java.lang.AssertionError: 
Expecting:
  <Service ActiveAuditManagerS3A in state ActiveAuditManagerS3A: STARTED, auditor=LoggingAuditor{ID='38e14ec9-3439-4b8f-a2f6-993514a2ee4a', headerEnabled=true, rejectOutOfSpan=false}}>
to be an instance of:
  <org.apache.hadoop.fs.s3a.audit.impl.NoopAuditManagerS3A>
but was instance of:
  <org.apache.hadoop.fs.s3a.audit.impl.ActiveAuditManagerS3A>    at org.apache.hadoop.fs.s3a.audit.ITestAuditManagerDisabled.testAuditorDisabled(ITestAuditManagerDisabled.java:60)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:748){noformat}