Repository: ansible/ansible-modules-extras

Issue:
maven_artifact cannot download sbt published artifact
##### Issue Type:

Please pick one and delete the rest:
- Bug Report
##### Plugin Name:

maven_artifact
##### Ansible Version:

```
ansible 2.0.0.2
  config file = /cygdrive/c/Users/.../MyProject/ansible.cfg
  configured module search path = Default w/o overrides
```
##### Ansible Configuration:
##### Environment:

N/A
##### Summary:

Unable to download a snapshot artifact from our company maven repo (Sonatype Nexus). The artifact was published with sbt and setting `publishMavenStyle := true`.

When publishing with sbt, it uploads an artifact with a "SNAPSHOT" suffix rather than a timestamp-build number suffix.

Sample maven-metadata.xml:

```
<metadata modelVersion="1.1.0">
  <groupId>[some-group-id]</groupId>
  <artifactId>[some-artifact-id]</artifactId>
  <version>0.1-SNAPSHOT</version>
  <versioning>
    <lastUpdated>20160223105322</lastUpdated>
  </versioning>
</metadata>
```
##### Steps To Reproduce:

Use maven-artifact to download any snapshot artifact from any maven repo.
##### Expected Results:

Artifact downloaded
##### Actual Results:

```
TASK [MyProject : Download artifact] ***************************
fatal: [local-dev-mwd]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible-tmp-1456157113.39-94202179773144/maven_artifact\", line 3089, in <module>\r\n    main()\r\n  File \"/tmp/ansible-tmp-1456157113.39-94202179773144/maven_artifact\", line 346, in main\r\n    if downloader.download(artifact, dest):\r\n  File \"/tmp/ansible-tmp-1456157113.39-94202179773144/maven_artifact\", line 237, in download\r\n    url = self.find_uri_for_artifact(artifact)\r\n  File \"/tmp/ansible-tmp-1456157113.39-94202179773144/maven_artifact\", line 202, in find_uri_for_artifact\r\n    timestamp = xml.xpath(\"/metadata/versioning/snapshot/timestamp/text()\")[0]\r\nIndexError: list index out of range\r\n", "msg": "MODULE FAILURE", "parsed": false}
```

maven_artifact cannot download sbt published artifact
##### Issue Type:

Please pick one and delete the rest:
- Bug Report
##### Plugin Name:

maven_artifact
##### Ansible Version:

```
ansible 2.0.0.2
  config file = /cygdrive/c/Users/.../MyProject/ansible.cfg
  configured module search path = Default w/o overrides
```
##### Ansible Configuration:
##### Environment:

N/A
##### Summary:

Unable to download a snapshot artifact from our company maven repo (Sonatype Nexus). The artifact was published with sbt and setting `publishMavenStyle := true`.

When publishing with sbt, it uploads an artifact with a "SNAPSHOT" suffix rather than a timestamp-build number suffix.

Sample maven-metadata.xml:

```
<metadata modelVersion="1.1.0">
  <groupId>[some-group-id]</groupId>
  <artifactId>[some-artifact-id]</artifactId>
  <version>0.1-SNAPSHOT</version>
  <versioning>
    <lastUpdated>20160223105322</lastUpdated>
  </versioning>
</metadata>
```
##### Steps To Reproduce:

Use maven-artifact to download any snapshot artifact from any maven repo.
##### Expected Results:

Artifact downloaded
##### Actual Results:

```
TASK [MyProject : Download artifact] ***************************
fatal: [local-dev-mwd]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible-tmp-1456157113.39-94202179773144/maven_artifact\", line 3089, in <module>\r\n    main()\r\n  File \"/tmp/ansible-tmp-1456157113.39-94202179773144/maven_artifact\", line 346, in main\r\n    if downloader.download(artifact, dest):\r\n  File \"/tmp/ansible-tmp-1456157113.39-94202179773144/maven_artifact\", line 237, in download\r\n    url = self.find_uri_for_artifact(artifact)\r\n  File \"/tmp/ansible-tmp-1456157113.39-94202179773144/maven_artifact\", line 202, in find_uri_for_artifact\r\n    timestamp = xml.xpath(\"/metadata/versioning/snapshot/timestamp/text()\")[0]\r\nIndexError: list index out of range\r\n", "msg": "MODULE FAILURE", "parsed": false}
```



Discussion:
<!-- @nayan92: Thanks for filing this bug report! -->

@chrisisbeef ping, this issue is waiting for your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_bug_new --->

@chrisisbeef, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

@nayan92 Can you verify this is still an issue?

I believe I have encountered this problem. I am using SBT to publish artifact to Artifactory maven repository. I am also using sbt-assembly https://github.com/sbt/sbt-assembly to add artifact with `assembly` classifier to my build & publish (deploy) procedure.

My `build.sbt` contains these modifications to achieve `sbt publish` publishing of `assembly` artifact:

```
artifact in (Compile, assembly) := {
  val art = (artifact in (Compile, assembly)).value
  art.copy(`classifier` = Some("assembly"))
}

addArtifact(artifact in (Compile, assembly), assembly)

publishMavenStyle := true
```

Version of my artifact is:

```
version := "1.0.0-SNAPSHOT"
```

When I hit

```
sbt publish
```

SBT first uploads artifacts to Maven repository as depicted in this `maven-metadata.xml`:

```
<?xml version="1.0" encoding="UTF-8"?>
<metadata>
   <groupId>sample_group</groupId>
   <artifactId>sample_artifact</artifactId>
   <version>1.0.0-SNAPSHOT</version>
   <versioning>
      <snapshot>
         <timestamp>20161004.084042</timestamp>
         <buildNumber>1</buildNumber>
      </snapshot>
      <lastUpdated>20161004084147</lastUpdated>
      <snapshotVersions>
         <snapshotVersion>
            <classifier>javadoc</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <classifier>sources</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <classifier>assembly</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <extension>pom</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
      </snapshotVersions>
   </versioning>
</metadata>
```

The files described in this `maven-metadata.xml` are present at Artifactory server and can be downloaded.
The URL for downloading `assembly` can be constructed using VERSION-TIMESTAMP-BUILDNUMBER. As this is done in https://github.com/ansible/ansible-modules-extras/blob/devel/packaging/language/maven_artifact.py#L215-L219

But then, when I relaunch `sbt publish`  to publish new artifact snapshot, I get this `maven-metadata.xml`:

```
<?xml version="1.0" encoding="UTF-8"?>
<metadata>
   <groupId>sample_group</groupId>
   <artifactId>sample_artifact</artifactId>
   <version>1.0.0-SNAPSHOT</version>
   <versioning>
      <snapshot>
         <timestamp>20161004.084718</timestamp>
         <buildNumber>2</buildNumber>
      </snapshot>
      <lastUpdated>20161004084720</lastUpdated>
      <snapshotVersions>
         <snapshotVersion>
            <classifier>javadoc</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084718-2</value>
            <updated>20161004084718</updated>
         </snapshotVersion>
         <snapshotVersion>
            <classifier>sources</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <classifier>assembly</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <extension>jar</extension>
            <value>1.0.0-20161004.084718-2</value>
            <updated>20161004084718</updated>
         </snapshotVersion>
         <snapshotVersion>
            <extension>pom</extension>
            <value>1.0.0-20161004.084718-2</value>
            <updated>20161004084718</updated>
         </snapshotVersion>
      </snapshotVersions>
   </versioning>
</metadata>
```

It seems like `assembly` and `sources` classifiers were not reuploaded / updated. But wait, if `jar` has changed and reuploaded, assembly had to change also. This well may be bug inside SBT and I am going to investigate. Further sbt publishing are continuing to keep the `buildNumber`  lag by one, so the problem does not fix by itself.

Nevermind the possibility of SBT bug, I have done testing of ansible `maven_artifact.py` correct behavior by trying to download this snapshot artifact using classic Maven 3.3.8. I have created `pom.xml` which depends on this artifact:

```
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>sample_group</groupId>
    <artifactId>sample_artifact_dependandant</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>sample_group</groupId>
            <artifactId>sample_artifact</artifactId>
            <version>[1.0.0-SNAPSHOT,)</version>
            <classifier>assembly</classifier>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.10</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>target</outputDirectory>
                            <overWriteReleases>false</overWriteReleases>
                            <overWriteSnapshots>false</overWriteSnapshots>
                            <overWriteIfNewer>true</overWriteIfNewer>
                            <excludeTransitive>true</excludeTransitive>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

        </plugins>
    </build>

    <repositories>
        <repository>
            <id>libssnapshot</id>
            <url>...</url>
        </repository>
        <repository>
            <id>libsrelease</id>
            <url>...</url>
        </repository>
    </repositories>
</project>
```

Running `mvn package` this should basically just download the dependency and put it to `target/` folder. And voila. Even with SBT publishing weirdly, plain maven can handle the situation and downloads `sample_artifact-1.0.0-20161004.084042-1-assembly.jar`.

So this brings me to the fix of `maven_artifact.py`, which should not only try to build the URL of artifact to download using ARTIFACT-VERSION-SNAPSHOT_TIMESTAMP-BUILDNUMBER-CLASSIFIER-EXTENSION, but should also scan `maven-metadata.xml` for the name of latest classifier + extension combination in case of snapshot artifact version.

<!-- @nayan92: Thanks for filing this bug report! -->

@chrisisbeef ping, this issue is waiting for your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_bug_new --->

@chrisisbeef, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

@nayan92 Can you verify this is still an issue?

I believe I have encountered this problem. I am using SBT to publish artifact to Artifactory maven repository. I am also using sbt-assembly https://github.com/sbt/sbt-assembly to add artifact with `assembly` classifier to my build & publish (deploy) procedure.

My `build.sbt` contains these modifications to achieve `sbt publish` publishing of `assembly` artifact:

```
artifact in (Compile, assembly) := {
  val art = (artifact in (Compile, assembly)).value
  art.copy(`classifier` = Some("assembly"))
}

addArtifact(artifact in (Compile, assembly), assembly)

publishMavenStyle := true
```

Version of my artifact is:

```
version := "1.0.0-SNAPSHOT"
```

When I hit

```
sbt publish
```

SBT first uploads artifacts to Maven repository as depicted in this `maven-metadata.xml`:

```
<?xml version="1.0" encoding="UTF-8"?>
<metadata>
   <groupId>sample_group</groupId>
   <artifactId>sample_artifact</artifactId>
   <version>1.0.0-SNAPSHOT</version>
   <versioning>
      <snapshot>
         <timestamp>20161004.084042</timestamp>
         <buildNumber>1</buildNumber>
      </snapshot>
      <lastUpdated>20161004084147</lastUpdated>
      <snapshotVersions>
         <snapshotVersion>
            <classifier>javadoc</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <classifier>sources</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <classifier>assembly</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <extension>pom</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
      </snapshotVersions>
   </versioning>
</metadata>
```

The files described in this `maven-metadata.xml` are present at Artifactory server and can be downloaded.
The URL for downloading `assembly` can be constructed using VERSION-TIMESTAMP-BUILDNUMBER. As this is done in https://github.com/ansible/ansible-modules-extras/blob/devel/packaging/language/maven_artifact.py#L215-L219

But then, when I relaunch `sbt publish`  to publish new artifact snapshot, I get this `maven-metadata.xml`:

```
<?xml version="1.0" encoding="UTF-8"?>
<metadata>
   <groupId>sample_group</groupId>
   <artifactId>sample_artifact</artifactId>
   <version>1.0.0-SNAPSHOT</version>
   <versioning>
      <snapshot>
         <timestamp>20161004.084718</timestamp>
         <buildNumber>2</buildNumber>
      </snapshot>
      <lastUpdated>20161004084720</lastUpdated>
      <snapshotVersions>
         <snapshotVersion>
            <classifier>javadoc</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084718-2</value>
            <updated>20161004084718</updated>
         </snapshotVersion>
         <snapshotVersion>
            <classifier>sources</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <classifier>assembly</classifier>
            <extension>jar</extension>
            <value>1.0.0-20161004.084042-1</value>
            <updated>20161004084042</updated>
         </snapshotVersion>
         <snapshotVersion>
            <extension>jar</extension>
            <value>1.0.0-20161004.084718-2</value>
            <updated>20161004084718</updated>
         </snapshotVersion>
         <snapshotVersion>
            <extension>pom</extension>
            <value>1.0.0-20161004.084718-2</value>
            <updated>20161004084718</updated>
         </snapshotVersion>
      </snapshotVersions>
   </versioning>
</metadata>
```

It seems like `assembly` and `sources` classifiers were not reuploaded / updated. But wait, if `jar` has changed and reuploaded, assembly had to change also. This well may be bug inside SBT and I am going to investigate. Further sbt publishing are continuing to keep the `buildNumber`  lag by one, so the problem does not fix by itself.

Nevermind the possibility of SBT bug, I have done testing of ansible `maven_artifact.py` correct behavior by trying to download this snapshot artifact using classic Maven 3.3.8. I have created `pom.xml` which depends on this artifact:

```
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>sample_group</groupId>
    <artifactId>sample_artifact_dependandant</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>sample_group</groupId>
            <artifactId>sample_artifact</artifactId>
            <version>[1.0.0-SNAPSHOT,)</version>
            <classifier>assembly</classifier>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.10</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>target</outputDirectory>
                            <overWriteReleases>false</overWriteReleases>
                            <overWriteSnapshots>false</overWriteSnapshots>
                            <overWriteIfNewer>true</overWriteIfNewer>
                            <excludeTransitive>true</excludeTransitive>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

        </plugins>
    </build>

    <repositories>
        <repository>
            <id>libssnapshot</id>
            <url>...</url>
        </repository>
        <repository>
            <id>libsrelease</id>
            <url>...</url>
        </repository>
    </repositories>
</project>
```

Running `mvn package` this should basically just download the dependency and put it to `target/` folder. And voila. Even with SBT publishing weirdly, plain maven can handle the situation and downloads `sample_artifact-1.0.0-20161004.084042-1-assembly.jar`.

So this brings me to the fix of `maven_artifact.py`, which should not only try to build the URL of artifact to download using ARTIFACT-VERSION-SNAPSHOT_TIMESTAMP-BUILDNUMBER-CLASSIFIER-EXTENSION, but should also scan `maven-metadata.xml` for the name of latest classifier + extension combination in case of snapshot artifact version.


Resolving patch:
diff --git a/packaging/language/maven_artifact.py b/packaging/language/maven_artifact.py
--- a/packaging/language/maven_artifact.py
+++ b/packaging/language/maven_artifact.py
@@ -216,6 +216,9 @@ def find_uri_for_artifact(self, artifact):
             xml = self._request(self.base + path, "Failed to download maven-metadata.xml", lambda r: etree.parse(r))
             timestamp = xml.xpath("/metadata/versioning/snapshot/timestamp/text()")[0]
             buildNumber = xml.xpath("/metadata/versioning/snapshot/buildNumber/text()")[0]
+            for snapshotArtifact in xml.xpath("/metadata/versioning/snapshotVersions/snapshotVersion"):
+                if len(snapshotArtifact.xpath("classifier/text()")) > 0 and snapshotArtifact.xpath("classifier/text()")[0] == artifact.classifier and len(snapshotArtifact.xpath("extension/text()")) > 0 and snapshotArtifact.xpath("extension/text()")[0] == artifact.extension:
+                    return self._uri_for_artifact(artifact, snapshotArtifact.xpath("value/text()")[0])
             return self._uri_for_artifact(artifact, artifact.version.replace("SNAPSHOT", timestamp + "-" + buildNumber))
 
         return self._uri_for_artifact(artifact, artifact.version)


Test patch: