Repository: bndtools/bnd
Pull request #6595: rework Add sources & calculateSourceAttachmentPath
Author: login: chrisrueger
Created: 2025-05-10T21:10:31Z | Merged: 2025-05-26T15:11:40Z

Closes https://github.com/bndtools/bnd/issues/6593

This PR does mainly 2 things:

- it removes a "hack" in `MavenBndRepository.get()` which was using the +-prefixed jar file (e.g. +somedependency-6.1.17.jar) which is created by "Add sources" in the repo browser. This had negative side effects because those +jars were put into the build output e.g. by `-includeresources`. 
- It improves the Eclipse Source attachment lookup to properly use bnds integrated mechanism to get a -sources.jar if it exists via the BSN_SOURCE_SUFFIX (bsn + ".source") mechanism

Note: I did not change the "Add sources" behavior. The +-jar with included sourcecode under OSGI-OPT is still created. I initially had it and it is reflected in the discussions below, but I reverted that. 

## What does it solve?

- for the initial issue https://github.com/bndtools/bnd/issues/6593 it solves the problem that other jars put inside my jar via `-includeresource` no longer contain the source code just because I used "Add sources" in the IDE
- this also makes the Eclipse IDE created jars more reproducible because they now should be the same as in a CI build




Diff:
diff --git a/biz.aQute.repository/src/aQute/bnd/repository/maven/provider/MavenBndRepository.java b/biz.aQute.repository/src/aQute/bnd/repository/maven/provider/MavenBndRepository.java
index 3c1e0c4687..c43fff3565 100644
--- a/biz.aQute.repository/src/aQute/bnd/repository/maven/provider/MavenBndRepository.java
+++ b/biz.aQute.repository/src/aQute/bnd/repository/maven/provider/MavenBndRepository.java
@@ -582,9 +582,6 @@ public File get(String bsn, Version version, Map<String, String> properties, fin
 		}
 
 		File f = storage.toLocalFile(archive);
-		File withSources = new File(f.getParentFile(), "+" + f.getName());
-		if (withSources.isFile() && withSources.lastModified() > f.lastModified())
-			f = withSources;
 
 		if (listeners.length == 0) {
 			return f;
diff --git a/bndtools.builder/src/org/bndtools/builder/classpath/BndContainerSourceManager.java b/bndtools.builder/src/org/bndtools/builder/classpath/BndContainerSourceManager.java
index 05cfe7f8b0..441dc33884 100644
--- a/bndtools.builder/src/org/bndtools/builder/classpath/BndContainerSourceManager.java
+++ b/bndtools.builder/src/org/bndtools/builder/classpath/BndContainerSourceManager.java
@@ -1,5 +1,10 @@
 package org.bndtools.builder.classpath;
 
+import static aQute.bnd.osgi.Constants.BSN_SOURCE_SUFFIX;
+import static aQute.bnd.osgi.Constants.VERSION_ATTR_LATEST;
+import static aQute.bnd.service.Strategy.EXACT;
+import static aQute.bnd.service.Strategy.HIGHEST;
+
 import java.io.File;
 import java.io.IOException;
 import java.io.InputStream;
@@ -11,6 +16,7 @@
 import java.util.Map;
 import java.util.Map.Entry;
 import java.util.Properties;
+import java.util.SortedSet;
 import java.util.jar.JarInputStream;
 import java.util.jar.Manifest;
 
@@ -34,6 +40,7 @@
 import aQute.bnd.header.Attrs;
 import aQute.bnd.osgi.Domain;
 import aQute.bnd.service.RepositoryPlugin;
+import aQute.bnd.service.Strategy;
 import aQute.bnd.version.Version;
 import aQute.lib.io.IO;
 import bndtools.central.Central;
@@ -160,28 +167,74 @@ private static File getSourceBundle(IPath path, Map<String, String> props, List<
 				return null;
 			}
 
+			String bsn = null;
+			String version = null;
+
 			Domain domain = Domain.domain(manifest);
 			Entry<String, Attrs> bsnAttrs = domain.getBundleSymbolicName();
-			if (bsnAttrs == null) {
+
+			if (bsnAttrs != null) {
+				bsn = bsnAttrs.getKey();
+				version = domain.getBundleVersion();
+			}
+
+			if (bsn == null) {
+				bsn = props.get("bsn");
+			}
+
+			if (bsn == null) {
 				return null;
 			}
-			String bsn = bsnAttrs.getKey();
-			String version = domain.getBundleVersion();
 
 			if (version == null) {
 				version = props.get("version");
 			}
 
+			if ("file".equals(version)) {
+				// this is a jar in the project folder and we cannot look that
+				// up in a repo
+				// we could only hope that this jar contains sources
+				// under OSGI-OPT/src and will be picked up by
+				// org.bndtools.builder.classpath.BndContainerInitializer.Updater.calculateSourceAttachmentPath(IPath,
+				// File)
+				return null;
+			}
+
+			String bsnSource = bsn + BSN_SOURCE_SUFFIX;
+			Strategy strategy = (version == null || VERSION_ATTR_LATEST.equals(version)) ? HIGHEST : EXACT;
+			Version v = null;
+
 			for (RepositoryPlugin repo : repositories) {
+
 				if (repo == null) {
 					continue;
 				}
+
 				if (repo instanceof WorkspaceRepository) {
 					continue;
 				}
-				File sourceBundle = repo.get(bsn + ".source", new Version(version), props);
-				if (sourceBundle != null) {
-					return sourceBundle;
+
+				if (HIGHEST == strategy) {
+					SortedSet<Version> vs = repo.versions(bsn);
+
+					if (vs != null && !vs.isEmpty()) {
+						Version latest = vs.last();
+						File sourceBundle = repo.get(bsnSource, latest, props);
+
+						if (sourceBundle != null) {
+							return sourceBundle;
+						}
+					}
+				} else {
+
+					if (v == null) {
+						v = new Version(version); // just parse once
+					}
+					File sourceBundle = repo.get(bsnSource, v, props);
+
+					if (sourceBundle != null) {
+						return sourceBundle;
+					}
 				}
 			}
 		} catch (final Exception e) {


Reviews:
user: login: reckart
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T04:52:36Z
user: login: chrisrueger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T07:33:25Z
user: login: chrisrueger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T07:33:43Z
user: login: chrisrueger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T16:54:15Z
user: login: chrisrueger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T17:30:15Z

Review comments:
diff_hunk: @@ -488,7 +488,42 @@ private void addProjectEntry(IPath path, List<IAccessRule> accessRules, List<ICl
 
 		private IPath calculateSourceAttachmentPath(IPath path, File file) {
 			JarInfo info = getJarInfo(file);
-			return info.hasSource ? path : null;
+			IPath sourcePath = info.hasSource ? path : null;
+
+			if (sourcePath == null) {
+				sourcePath = trySourcesJar(file);
+			}
+
+			return sourcePath;
+		}
+
+		/**
+		 * try to see if there is file bsn-sources.jar e.g. as downloaded by the
+		 * Maven repos
+		 *
+		 * @param file
+		 * @return the bsn-sources.jar file if found, otherwise
+		 *         <code>null</code>
+		 */
+		private IPath trySourcesJar(File file) {
+
+			if (file.getName()
+				.endsWith(".jar")) {
+
+				try {
+					File withSources = new File(file.getParentFile(), file.getName()
+						.substring(0, file.getName()
+							.indexOf(".jar"))
path: bndtools.builder/src/org/bndtools/builder/classpath/BndContainerInitializer.java
user: login: reckart
body: Should probably be a `lastIndexOf`. How about using a method like Apache Commons IO `FilenameUtils.getExtension()`?
created_at: 2025-05-13T04:48:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -488,7 +488,42 @@ private void addProjectEntry(IPath path, List<IAccessRule> accessRules, List<ICl
 
 		private IPath calculateSourceAttachmentPath(IPath path, File file) {
 			JarInfo info = getJarInfo(file);
-			return info.hasSource ? path : null;
+			IPath sourcePath = info.hasSource ? path : null;
+
+			if (sourcePath == null) {
+				sourcePath = trySourcesJar(file);
+			}
+
+			return sourcePath;
+		}
+
+		/**
+		 * try to see if there is file bsn-sources.jar e.g. as downloaded by the
+		 * Maven repos
+		 *
+		 * @param file
+		 * @return the bsn-sources.jar file if found, otherwise
+		 *         <code>null</code>
+		 */
+		private IPath trySourcesJar(File file) {
+
+			if (file.getName()
+				.endsWith(".jar")) {
path: bndtools.builder/src/org/bndtools/builder/classpath/BndContainerInitializer.java
user: login: reckart
body: Should maybe also allow for `.JAR`. File names are not case sensitive on all platforms.
created_at: 2025-05-13T04:49:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -488,7 +488,42 @@ private void addProjectEntry(IPath path, List<IAccessRule> accessRules, List<ICl
 
 		private IPath calculateSourceAttachmentPath(IPath path, File file) {
 			JarInfo info = getJarInfo(file);
-			return info.hasSource ? path : null;
+			IPath sourcePath = info.hasSource ? path : null;
+
+			if (sourcePath == null) {
+				sourcePath = trySourcesJar(file);
+			}
+
+			return sourcePath;
+		}
+
+		/**
+		 * try to see if there is file bsn-sources.jar e.g. as downloaded by the
+		 * Maven repos
+		 *
+		 * @param file
+		 * @return the bsn-sources.jar file if found, otherwise
+		 *         <code>null</code>
+		 */
+		private IPath trySourcesJar(File file) {
+
+			if (file.getName()
+				.endsWith(".jar")) {
+
+				try {
+					File withSources = new File(file.getParentFile(), file.getName()
+						.substring(0, file.getName()
+							.indexOf(".jar"))
+						+ "-sources.jar");
+
+					if (withSources.isFile()) {
+						return fileToPath(withSources);
+					}
+				} catch (Exception e) {
+					// ignore
path: bndtools.builder/src/org/bndtools/builder/classpath/BndContainerInitializer.java
user: login: reckart
body: A trace-level logging might be helpful here for possible debugging cases and would remove the need to actually fire up a debugger.
created_at: 2025-05-13T04:51:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -21,42 +22,17 @@ public static void addSources(final Archive archive, Function<Archive, Promise<F
 
 		if (pBinary.getFailure() == null) {
 			final File binary = pBinary.getValue();
-			final File out = new File(binary.getParentFile(), "+" + binary.getName());
-
-			if (!out.isFile()) {
-				Archive a = archive.revision.archive("jar", "sources");
-				Promise<File> pSources = lookup.apply(a);
-
-				if (pSources.getFailure() == null) {
-					final File sources = pSources.getValue();
-
-					if (sources.isFile() && sources.length() > 1000) {
-						map.put("Add Sources", () -> {
-
-							try {
-
-								try (Jar src = new Jar(sources)) {
-
-									try (Jar bin = new Jar(binary)) {
-										bin.setDoNotTouchManifest();
-										for (String path : src.getResources()
-											.keySet())
-											bin.putResource("OSGI-OPT/src/" + path, src.getResource(path));
-										bin.write(out);
-									}
-									out.setLastModified(System.currentTimeMillis());
-								}
-							} catch (Exception e) {
-								throw Exceptions.duck(e);
-							}
-						});
-						return;
-					}
-				}
-			}
+
+			map.put("Add Sources", () -> {
+
+				Archive a = archive.getOther(JAR_EXTENSION, SOURCES_CLASSIFIER);
+				lookup.apply(a);
+
+			});
+			return;
 		}
 
-		map.put("-Add Sources", null);
+		// map.put("-Add Sources", null);
path: biz.aQute.repository/src/aQute/bnd/repository/maven/util/RepoActionsUtil.java
user: login: reckart
body: Is this commented out line still necessary? If yes, a comment explaining its use would be helpful.
created_at: 2025-05-13T04:51:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -488,7 +488,42 @@ private void addProjectEntry(IPath path, List<IAccessRule> accessRules, List<ICl
 
 		private IPath calculateSourceAttachmentPath(IPath path, File file) {
 			JarInfo info = getJarInfo(file);
-			return info.hasSource ? path : null;
+			IPath sourcePath = info.hasSource ? path : null;
+
+			if (sourcePath == null) {
+				sourcePath = trySourcesJar(file);
+			}
+
+			return sourcePath;
+		}
+
+		/**
+		 * try to see if there is file bsn-sources.jar e.g. as downloaded by the
+		 * Maven repos
+		 *
+		 * @param file
+		 * @return the bsn-sources.jar file if found, otherwise
+		 *         <code>null</code>
+		 */
+		private IPath trySourcesJar(File file) {
+
+			if (file.getName()
+				.endsWith(".jar")) {
+
+				try {
+					File withSources = new File(file.getParentFile(), file.getName()
+						.substring(0, file.getName()
+							.indexOf(".jar"))
path: bndtools.builder/src/org/bndtools/builder/classpath/BndContainerInitializer.java
user: login: chrisrueger
body: Thanks. I'll try to improve it. FilenameUtils is currently not a dependency at the moment and I don't want to introduce one just for this. 
created_at: 2025-05-13T07:33:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -488,7 +488,42 @@ private void addProjectEntry(IPath path, List<IAccessRule> accessRules, List<ICl
 
 		private IPath calculateSourceAttachmentPath(IPath path, File file) {
 			JarInfo info = getJarInfo(file);
-			return info.hasSource ? path : null;
+			IPath sourcePath = info.hasSource ? path : null;
+
+			if (sourcePath == null) {
+				sourcePath = trySourcesJar(file);
+			}
+
+			return sourcePath;
+		}
+
+		/**
+		 * try to see if there is file bsn-sources.jar e.g. as downloaded by the
+		 * Maven repos
+		 *
+		 * @param file
+		 * @return the bsn-sources.jar file if found, otherwise
+		 *         <code>null</code>
+		 */
+		private IPath trySourcesJar(File file) {
+
+			if (file.getName()
+				.endsWith(".jar")) {
path: bndtools.builder/src/org/bndtools/builder/classpath/BndContainerInitializer.java
user: login: chrisrueger
body: Good point. 
created_at: 2025-05-13T07:33:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -21,42 +22,17 @@ public static void addSources(final Archive archive, Function<Archive, Promise<F
 
 		if (pBinary.getFailure() == null) {
 			final File binary = pBinary.getValue();
-			final File out = new File(binary.getParentFile(), "+" + binary.getName());
-
-			if (!out.isFile()) {
-				Archive a = archive.revision.archive("jar", "sources");
-				Promise<File> pSources = lookup.apply(a);
-
-				if (pSources.getFailure() == null) {
-					final File sources = pSources.getValue();
-
-					if (sources.isFile() && sources.length() > 1000) {
-						map.put("Add Sources", () -> {
-
-							try {
-
-								try (Jar src = new Jar(sources)) {
-
-									try (Jar bin = new Jar(binary)) {
-										bin.setDoNotTouchManifest();
-										for (String path : src.getResources()
-											.keySet())
-											bin.putResource("OSGI-OPT/src/" + path, src.getResource(path));
-										bin.write(out);
-									}
-									out.setLastModified(System.currentTimeMillis());
-								}
-							} catch (Exception e) {
-								throw Exceptions.duck(e);
-							}
-						});
-						return;
-					}
-				}
-			}
+
+			map.put("Add Sources", () -> {
+
+				Archive a = archive.getOther(JAR_EXTENSION, SOURCES_CLASSIFIER);
+				lookup.apply(a);
+
+			});
+			return;
 		}
 
-		map.put("-Add Sources", null);
+		// map.put("-Add Sources", null);
path: biz.aQute.repository/src/aQute/bnd/repository/maven/util/RepoActionsUtil.java
user: login: chrisrueger
body: I'll remove that too.
created_at: 2025-05-13T16:54:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -488,7 +488,42 @@ private void addProjectEntry(IPath path, List<IAccessRule> accessRules, List<ICl
 
 		private IPath calculateSourceAttachmentPath(IPath path, File file) {
 			JarInfo info = getJarInfo(file);
-			return info.hasSource ? path : null;
+			IPath sourcePath = info.hasSource ? path : null;
+
+			if (sourcePath == null) {
+				sourcePath = trySourcesJar(file);
+			}
+
+			return sourcePath;
+		}
+
+		/**
+		 * try to see if there is file bsn-sources.jar e.g. as downloaded by the
+		 * Maven repos
+		 *
+		 * @param file
+		 * @return the bsn-sources.jar file if found, otherwise
+		 *         <code>null</code>
+		 */
+		private IPath trySourcesJar(File file) {
+
+			if (file.getName()
+				.endsWith(".jar")) {
path: bndtools.builder/src/org/bndtools/builder/classpath/BndContainerInitializer.java
user: login: chrisrueger
body: I added support for case-insensitive file extension. Although I read that the convention is all lowercase, also enforced by maven central, but unfortunatelly it does not seem to be specified. So I assume there could be uppercase .JAR files out there in enterprise maven repos.
created_at: 2025-05-13T17:30:15Z
author_association: CONTRIBUTOR

PR comments:
user: login: chrisrueger
created_at: 2025-05-13T17:34:13Z
body: Anybody from the longtime committers with knowledge about the history of `+filename.jar` files? @pkriens @bjhargrave @kriegfrj Any objections? 
author_association: CONTRIBUTOR
user: login: bjhargrave
created_at: 2025-05-13T18:53:55Z
body: > Anybody from the longtime committers with knowledge about the history of `+filename.jar` files?

As far as I can recall, they are just source-enriched (source code added to OSGI-OPT) versions of the original jar so Bnd did not have to manage -sources.jar files.
author_association: MEMBER
user: login: bjhargrave
created_at: 2025-05-13T18:54:57Z
body: This PR seems to be undoing the original goal of a single jar with classes and source.
author_association: MEMBER
user: login: chrisrueger
created_at: 2025-05-13T19:31:39Z
body: > As far as I can recall, they are just source-enriched (source code added to OSGI-OPT) versions of the original jar so Bnd did not have to manage -sources.jar files.

This is my impression too.

> This PR seems to be undoing the original goal of a single jar with classes and source.

Correct. 
UPDATE: see my [comment below](https://github.com/bndtools/bnd/pull/6595#issuecomment-2878727344). If the main .jar already contains the sources in `OSGI-OPT` then this is used. This behavior is unchanged.

Now bndtools can use the `-sources.jar` which is downloaded by the MavenBndRepository and BndPomRepository anyway, and does not create a copy of the original .jar + sourcecode anymore .

Is this a problem? What was the reason in the past to not use the  `-sources.jar`? Was it only for the Eclipse-Source-Lookup? 

(just for clarification: I am not undoing the [-sources](https://bnd.bndtools.org/instructions/sources.html) instruction which puts source code into the generated .jar under OSGI-OPT. This is untouched, thought it might sound related) 




author_association: CONTRIBUTOR
user: login: kriegfrj
created_at: 2025-05-14T02:52:05Z
body: > > As far as I can recall, they are just source-enriched (source code added to OSGI-OPT) versions of the original jar so Bnd did not have to manage -sources.jar files.
> 
> This is my impression too.
> 
> > This PR seems to be undoing the original goal of a single jar with classes and source.
> 
> Correct.
> 
> Now bndtools can use the `-sources.jar` which is downloaded by the MavenBndRepository and BndPomRepository anyway, and does not create a copy of the original .jar + sourcecode anymore .
> 
> Is this a problem? What was the reason in the past to not use the `-sources.jar`? Was it only for the Eclipse-Source-Lookup?
> 

I don't have a time for a deep-dive, but my initial concern is to make sure that the source lookup will continue to work after this change. It could be that the `BndContainerInitializer` might need some re-working to attach the sources to the `IClasspathEntry`s that it creates, which (from my very old and fading memory) it doesn't currently do - it relies on them being present in the binary jar in `OSGI-OPT`. You might also need a hook to refresh this when you download sources from the MavenBndRepository context menu (perhaps an option of the MavenBndRepository config to automatically download sources alongside the binaries, ideally do it asychronously).

Also need to see how this change will interplay with the P2Repository.
author_association: CONTRIBUTOR
user: login: reckart
created_at: 2025-05-14T04:52:48Z
body: Maybe it would be a good idea to use the legacy JAR-with-source as the default and if there are no sources in the JAR, then check if there is a Maven-style `-sources` JAR which could be used?
author_association: CONTRIBUTOR
user: login: chrisrueger
created_at: 2025-05-14T05:32:26Z
body: > Maybe it would be a good idea to use the legacy JAR-with-source as the default and if there are no sources in the JAR, then check if there is a Maven-style `-sources` JAR which could be used?

That is exactly how it now works. I forgot to mention that I left the existing lookup of OSGI-OPT/src in place. So this is done first. Only if no sources are found inside the jar, I do the new lookup in this PR. @reckart @kriegfrj @bjhargrave 

So for example the OSGi core jars have sources under OSGI-OPT, and for them the sources inside the jar are used.
Only if the jar does not contain sources, I now try to see if there is a -sources.jar (added via Repoviewer Right-Click / Add-Sources), and if yes take this. 
author_association: CONTRIBUTOR
user: login: chrisrueger
created_at: 2025-05-22T07:47:19Z
body: > * this also makes the Eclipse IDE created jars more reproducible because they now should be the same as in a CI build

I just want to highlight this **reproducibility** fact which might appear less important at first glance, but in fact it is important IMO

I was bitten by that again. 
I just noticed that a local gradle build output executable jar (using bnd gradle plugin) was much larger than the build on our CI server. 
After unpacking some the executable jar and some jars in there I noticed that it was because my local maven repository (`~/.m2`)  contained the `+` jar files because of `-includeresource`. This happened because ~/.m2 contained such files, because I did "Add sources" in Eclipse for some of those jars which got included. 

<img width="406" alt="image" src="https://github.com/user-attachments/assets/ca13fdfb-dd10-43f4-b9a0-ac5ac6a7d18c" />

In other words: a reproducible build is difficult with this situation, because it now heavily depends on the environment and whether or not there are `+` jar files in the local maven repo or not. 

So I hope that this supports the case for this PR.
author_association: CONTRIBUTOR
user: login: chrisrueger
created_at: 2025-05-23T19:56:15Z
body: Thanks @pkriens for the tip with the magic `BSN_SOURCE_SUFFIX` (bsn + ".source"). I think I have a better approach now. 
You were right that a lot was already there. But it had a problem. It only worked with a real OSGi bundle which had a proper manifest with `BundleSymbolicName`. In our case we have some non-OSGI jars but with BSN/GAV. This information was already passed in a Map. I use  that now for non-OSGI jars. That way I managed to get sources for more jars. 

It also supports two different strategies (HIGHEST and EXACT), because some bundles are referenced as 'latest' instead of an exact version. 
This logic is a little bit borrowed from `Project.getBundle()` but not completely because I need to do less.

I also added back the "Add Sources" stuff which creates the `+jar-file` in case somebody needs it. But the Eclipse Source attachment lookup now does not get it in its hand anymore and instead tries to lookup the sources jar using the repo via the `BSN_SOURCE_SUFFIX`
author_association: CONTRIBUTOR
user: login: chrisrueger
created_at: 2025-05-26T15:14:17Z
body: For all involved here in the discussion:
I decided to merge this, after talking to @pkriens and approaching another strategy. So most of the discussions above is a bit outdated, since the change is much smaller than I had initially.

I have used the current code for a day with no issues so far and I will continue monitoring it. If I discover problems I will try to address them quickly. 

Please try out the next SNAPSHOT and let me know of any issues. 
author_association: CONTRIBUTOR