av9ash/GitBugs huggingface dataset:
Bug: [JDK 16] TestRPC#testAuthorization fails by ClassCastException
Status: Open | Priority: Major | Resolution: 
Created: 09/Apr/21 09:54

{code}

[ERROR] testAuthorization(org.apache.hadoop.ipc.TestRPC)  Time elapsed: 1.066 s  <<< ERROR!
java.lang.ClassCastException: class java.net.SocketException cannot be cast to class org.apache.hadoop.ipc.RemoteException (java.net.SocketException is in module java.base of loader 'bootstrap'; org.apache.hadoop.ipc.RemoteException is in unnamed module of loader 'app')
	at org.apache.hadoop.ipc.TestRPC.doRPCs(TestRPC.java:591)
	at org.apache.hadoop.ipc.TestRPC.testAuthorization(TestRPC.java:639)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
Bug: Remove lock contention in overlay of Configuration
Status: Open | Priority: Major | Resolution: 
Created: 09/Jun/21 09:31

The *overlay* field of class *Configuration* is a point of lock contention, which is bad for performance.

E.g.,
{code:java}
$ grep 'waiting to lock <0x00007fa4fc113378>' 17326.jstack | uniq -c
 257 - waiting to lock <0x00007fa4fc113378> (a org.apache.hadoop.conf.Configuration)
{code}
and the thread stack is as follows:
{code:java}
"hconnection-0x66971f6b-shared--pool1-t1060" #6315 daemon prio=5 os_prio=0 tid=0x00007f5c04018800 nid=0x11f31 waiting for monitor entry [0x00007f567f3f4000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.conf.Configuration.getOverlay(Configuration.java:1328)
        - waiting to lock <0x00007fa4fc113378> (a org.apache.hadoop.conf.Configuration)
        at org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684)
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088)
        at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
        at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1375)
        at org.apache.hadoop.hbase.ipc.PhoenixRpcSchedulerFactory.getMetadataPriority(PhoenixRpcSchedulerFactory.java:92)
        at org.apache.hadoop.hbase.ipc.controller.MetadataRpcController.<init>(MetadataRpcController.java:59)
        at org.apache.hadoop.hbase.ipc.controller.ClientRpcControllerFactory.getController(ClientRpcControllerFactory.java:57)
        at org.apache.hadoop.hbase.ipc.controller.ClientRpcControllerFactory.newController(ClientRpcControllerFactory.java:41)
        at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:216)
        at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:65)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:365)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:339)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
        at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{code}
Bug: Abfs directory delete times out on large directory tree w/ Oauth: OperationTimedOut
Status: Open | Priority: Major | Resolution: 
Created: 11/May/21 12:15
Affects Version/s: 3.3.0

Timeouts surfacing on abfs when a delete of a large directory tree is invoked.
{code}
StatusDescription=Operation could not be completed within the specified time.
ErrorCode=OperationTimedOut
ErrorMessage=Operation could not be completed within the specified time.
{code}

This has surfaced in v1 FileOutputCommitter cleanups, implying the directories created there (many many dirs, no files remaining after the job commit) is sufficient to create the problem.
Bug: ABFS: ITestAzureBlobFileSystemLease test failure for Appendblob HNS OAuth
Status: Open | Priority: Major | Resolution: 
Created: 30/Jun/21 09:03
Affects Version/s: 3.3.1

Test testTwoWritersCreateAppendNoInfiniteLease fails with error:

java.io.IOException: Operation failed: "Value for one of the query parameters specified in the request URI is invalid."
Bug: Update commons-io to 2.8.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/21 08:44
Affects Version/s: 3.3.1

https://nvd.nist.gov/vuln/detail/CVE-2021-29425

In Apache Commons IO before 2.7, When invoking the method FileNameUtils.normalize with an improper input string, like "//../foo", or "\\..\foo", the result would be the same value, thus possibly providing access to files in the parent directory, but not further above (thus "limited" path traversal), if the calling code would use the result to construct a path value.

We don't use this API in the Hadoop code, but it's still good to update anyway (we're on 2.5, which is 4 years old)
Bug: Remove any rocksdb exclusion code
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/21 03:16
Affects Version/s: 3.4.0

RocksDB was added as part of Ozone runtime, and we added code to exclude RocksDB from the core runtime. Now that Ozone is a separate project, we should remove anything referencing RocksDB now.
Bug: Upgrade JUnit to 4.13.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/21 17:24
Affects Version/s: 2.10.2

JUnit 4.13.1 has a bug that is reported in Junit [issue-1652|https://github.com/junit-team/junit4/issues/1652] _Timeout ThreadGroups should not be destroyed_

After upgrading Junit to 4.13.1 in HADOOP-17602, {{TestBlockRecovery}}  started to fail regularly in branch-3.x and branch-2.10.
While investigating the failure in branch-2.10 HDFS-16072, I found out that the bug is the main reason {{TestBlockRecovery}}  started to fail because the timeout of the Junit would try to close a ThreadGroup that has been already closed which throws the {{java.lang.IllegalThreadStateException}}.

The bug has been fixed in Junit-4.13.2

For branch-3.x, HDFS-15940 did not address the root cause of the problem. Eventually, Splitting the {{TestBlockRecovery}} hid the bug, but the upgrade needs to be done so that the problem does not show up in another unit test.
Bug: checkcompatibility.py errors out when specifying annotations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/21 09:20
Affects Version/s: 3.3.1

[https://github.com/apache/hadoop/blob/trunk/dev-support/bin/checkcompatibility.py#L178]
{code:java}
 with file(annotations_path, "w") as f: {code}
is not a valid Pythonic code.
Bug: SequenceFile.Writer should implement StreamCapabilities
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/21 19:03
Affects Version/s: 3.3.1

Following exception is thrown whenever we invoke ProtoMessageWriter.hflush on S3 from Tez, which internally calls org.apache.hadoop.io.SequenceFile$Writer.hflush ->  org.apache.hadoop.fs.FS DataOutputStream.hflush -> S3ABlockOutputStream.hflush which is not implemented and throws java.lang.UnsupportedOperationException. 

bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler" level="ERROR" thread="HistoryEventHandlingThread"] Thread Thread[HistoryEventHandlingThread, 5,main] threw an Exception.^Mjava.lang.UnsupportedOperationException: S3A streams are not Syncable^M at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M at org.apache.hadoop.fs.FS DataOutputStream.hflush(FSDataOutputStream.java:136)^M at org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M at org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr iter.java:64)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han dleEvent(ProtoHistoryLoggingService.java:198)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M at java.lang.Thread.run(Thread.java:748)^M

In order to fix this issue we should implement StreamCapabilities in SequenceFile.Writer. Also, we should fall back to flush(), if hflush() is not supported.
Bug: WASB : Make metadata checks case insensitive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/21 06:11
Affects Version/s: 2.7.0

WASB driver uses meta data on blobs to denote permission, whether its a place holder 0 sized blob for dir etc.
For storage migration users uses Azcopy, it copies the blobs but will cause the metadata keys to get changed to camel case. As per discussion with MSFT Azcopy team, this is a known issue and technical limitation.  This is what Azcopy team explained
"For context, blob metadata is implemented with HTTP headers. They are case insensitive but case preserving. 
There is a known issue with the Go language. The HTTP client that it provides does this case modification to the response headers before we can read the raw values, so the destination metadata keys have a different casing than the source. We’ve reached out to the Go Team in the past but weren’t successful in convincing them to change the behaviour. We don’t have a short term solution right now"

So propose to change the metadata key checks to do case insensitive checks.  May be make case insensitive check configurable with defaults to false for compatibility.
Bug: WASB : Support disabling buffered reads in positional reads
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/21 05:58
Affects Version/s: 3.3.2

This is just like HADOOP-17038

Right now it will do a seek to the position , read and then seek back to the old position. (As per the impl in the super class)
In HBase kind of workloads we rely mostly on short preads. (like 64 KB size by default). So would be ideal to support a pure pos read API which will not even keep the data in a buffer but will only read the required data as what is asked for by the caller. (Not reading ahead more data as per the read size config)

Allow an optional boolean config to be specified while opening file for read using which buffered pread can be disabled.
FutureDataInputStreamBuilder openFile(Path path)
Bug: Remove an invalid comment content in the FileContext class
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/21 09:55
Affects Version/s: 3.4.0

There is an invalid comment content in the FileContext class:
   /**
    * Protected Static Factory methods for getting a FileContexts
    * that take a AbstractFileSystem as input. To be used for testing.
    */
It doesn't seem to have any effect.
Bug: Update clover-maven-plugin version from 3.3.0 to 4.4.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jun/21 11:28
Affects Version/s: 3.3.2

Update clover-maven-plugin version to 4.4.1, some important changes are as follows:
 * the license key is no longer required to run Clover
 * the Atlassian brand and logos were removed to avoid any trademark violations
 * the 'org.openclover.*' groupId is used for artifacts

The license key is no longer required is the key factor to do this update. So the developer can run and get the code coverage easier.
Bug: [thirdparty] Do not exclude error_prone
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/21 15:48
Affects Version/s: 3.3.1

Building Ozone on Hadoop 3.3.1-RC1, found the following error during compilation:

{noformat}
18:40:02 2021/05/24 10:40:02 INFO    : [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-hdds-common: Compilation failure
18:40:02 2021/05/24 10:40:02 INFO    : [ERROR] cannot access org.apache.hadoop.thirdparty.com.google.errorprone.annotations.CanIgnoreReturnValue
18:40:02 2021/05/24 10:40:02 INFO    : [ERROR] class file for org.apache.hadoop.thirdparty.com.google.errorprone.annotations.CanIgnoreReturnValue not found
18:40:02 2021/05/24 10:40:02 INFO    : [ERROR] Consult the following stack trace for details.
18:40:02 2021/05/24 10:40:02 INFO    : [ERROR] com.sun.tools.javac.code.Symbol$CompletionFailure: class file for org.apache.hadoop.thirdparty.com.google.errorprone.annotations.CanIgnoreReturnValue not found
18:40:02 2021/05/24 10:40:02 INFO    : [ERROR] -> [Help 1]
{noformat}

The error_prone was excluded by YARN-10195 to avoid dependency divergence with YARN Timeline service. However, now that Hadoop's guava is shaded, we shouldn't have problem with diverging dependencies. So let's add it back.
Bug: Update the ubuntu version in the build instruction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/21 10:55
Affects Version/s: 3.4.0

In BUILDING.txt
{noformat}
Installing required packages for clean install of Ubuntu 14.04 LTS Desktop:
{noformat}
Ubuntu 14 is already EoL, should be updated to 18 or 20.
Bug: Remove appender EventCounter to avoid instantiation 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Apr/21 19:17
Affects Version/s: 3.4.0

After removal of EventCounter class, we are not able to bring up HDFS cluster.
{code:java}
log4j:ERROR Could not instantiate class [org.apache.hadoop.log.metrics.EventCounter].
java.lang.ClassNotFoundException: org.apache.hadoop.log.metrics.EventCounter
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
	at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
	at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
	at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
	at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
	at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:229)
	at org.apache.hadoop.hdfs.tools.GetConf.<clinit>(GetConf.java:131)
log4j:ERROR Could not instantiate appender named "EventCounter".
{code}
We need to clean up log4j.properties to avoid instantiating appender EventCounter.
Bug: Restrict imports from org.apache.curator.shaded
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/21 19:19
Affects Version/s: 3.3.1

Once HADOOP-17653 gets in, we should ban "org.apache.curator.shaded" imports as discussed on PR#2945. We can use enforcer-rule to restrict imports such that if ever used, mvn build fails.

Thanks for the suggestion [~weichiu] [~aajisaka] [~stevel@apache.org]
Bug: Avoid Potential NPE in org.apache.hadoop.fs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/21 03:48
Affects Version/s: 3.4.0

https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/
 Hello,
Our code analyses found the following potential NPE:

 
{code:java}
  public Path getParent() {
    String path = uri.getPath();
    int lastSlash = path.lastIndexOf('/');
    int start = startPositionWithoutWindowsDrive(path);
    if ((path.length() == start) ||               // empty path
        (lastSlash == start && path.length() == start+1)) { // at root
      return null;
    }
{code}
 
{code:java}
  public FSDataOutputStream createInternal (Path f,
      EnumSet<CreateFlag> flag, FsPermission absolutePermission, int bufferSize,
      short replication, long blockSize, Progressable progress,
      ChecksumOpt checksumOpt, boolean createParent) throws IOException {
    checkPath(f);
    
    // Default impl assumes that permissions do not matter
    // calling the regular create is good enough.
    // FSs that implement permissions should override this.    if (!createParent) { // parent must exist.
      // since this.create makes parent dirs automatically
      // we must throw exception if parent does not exist.
      final FileStatus stat = getFileStatus(f.getParent()); // NPE!
      if (stat == null) {
        throw new FileNotFoundException("Missing parent:" + f);
      }
{code}
Full Trace:

1. Return null to caller
https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java#L432

2. The return value of function getParent is used as the 1st parameter in function getFileStatus (the return value of function getParent can be null)
https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java#L93

3. f is used as the 1st parameter in function checkPath (f can be null)
https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java#L127

4. path is passed as the this pointer to function toUri (path can be null)
https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java#L369

Commit: f40e3eb0590f85bb42d2471992bf5d524628fdd6
Bug: Remove hardcoded SunX509 usage from SSLFactory
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/21 00:05
Affects Version/s: 3.4.0

In SSLFactory.SSLCERTIFICATE, used by FileBasedKeyStoresFactory and ReloadingX509TrustManager, there is a hardcoded reference to "SunX509" which is used to get a KeyManager/TrustManager. This KeyManager type might not be available if using the other JSSE providers, e.g.,  in FIPS deployment.

 
{code:java}
WARN org.apache.hadoop.hdfs.web.URLConnectionFactory: Cannot load customized ssl related configuration. Fall
 back to system-generic settings.
 java.security.NoSuchAlgorithmException: SunX509 KeyManagerFactory not available
 at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
 at javax.net.ssl.KeyManagerFactory.getInstance(KeyManagerFactory.java:137)
 at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:186)
 at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:187)
 at org.apache.hadoop.hdfs.web.SSLConnectionConfigurator.<init>(SSLConnectionConfigurator.java:50)
 at org.apache.hadoop.hdfs.web.URLConnectionFactory.getSSLConnectionConfiguration(URLConnectionFactory.java:100)
 at org.apache.hadoop.hdfs.web.URLConnectionFactory.newDefaultURLConnectionFactory(URLConnectionFactory.java:79)
{code}
This ticket is opened to use the DefaultAlgorithm defined by Java system property: 

ssl.KeyManagerFactory.algorithm and ssl.TrustManagerFactory.algorithm.
Bug: Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/21 10:59
Affects Version/s: 3.4.0

Now that all sub-tasks to remove dependency on Guava Sets are completed, we should move restrict-imports-enforcer-rule for Guava Sets import in hadoop-main pom and remove from individual project poms.