Repository: VeryGoodOpenSource/very_good_cli

Issue:
fix: Reporting of malformed licenses
**Description**
`very_good packages check licenses` fails to report pubs with unknown licenses (e.g. [rxdart 0.27.7](https://pub.dev/packages/rxdart/versions/0.27.7)):
```bash
$ very_good packages check licenses
✓ Retrieved 0 licenses from 1 package. (0.7s)
```

**Steps To Reproduce**

1. Clone [example repo](https://github.com/dominicmh/license_check) (includes `rxdart` dependency in `pubspec.lock`)
2. Execute `dart pub global activate very_good_cli; very_good packages check licenses`

**Expected Behavior**
`✓ Retrieved 1 license from 1 package of type: unknown (1).`

**Actual Behavior**
`✓ Retrieved 0 licenses from 1 package.`



Discussion: Hi @dominicmh thanks for opening an issue! And sorry for the late reply, I haven't been able to catch-up recently.

I've been able to reproduce this issue by using the following `pubspec.yaml`.

<details>

<summary> puspec.yaml </summary>

```yaml
name: example
description: An example project.
version: 1.0.0+1
publish_to: none

environment:
  sdk: "^3.3.0"

dependencies:
  rxdart: ^0.27.7
```
</details>

@dominicmh are you interested in contributing with a fix? If so, let me know and I can assign you to this item and assist you on landing it 🙌 
Hi @alestiago – I can't really promise any delivery date, could be weeks or even months. But generally I'm open to fix this.
Hi, following PR introduced this issue: https://github.com/VeryGoodOpenSource/very_good_cli/pull/883
So you can use version 0.17.1 until it is fixed. Maybe i can look into this issue :)
I analyzed the issue und added a [failing test](https://github.com/dominicmh/very_good_cli/commit/2ec4fc4920a9c93dc37d2ca1bf750f5fc017c7a4) for the bug.

# Analysis

- PANA's `detectLicense` is used to parse all detected license types from the LICENSE file (https://github.com/VeryGoodOpenSource/very_good_cli/commit/2ec4fc4920a9c93dc37d2ca1bf750f5fc017c7a4)
- Text from the LICENSE file that doesn't match to a standardized license format is simply ignored in the returned result.
- The answer what to do with ignored text is not straight-foward. Options:
  1. If there is a LICENSE file, but the result doesn't include any matched license types (`Result.matches.isEmpty`), add license type `SpdxLicense.$unknown.value` for the given dependency
  2. If some proportion of the LICENSE file's text cannot be matched to a license type, add (an additional) license type `SpdxLicense.$unknown.value` for the given dependency. This proportion can be retrieved through `Result.unclaimedTokenPercentage` and a threshold would need to be defined. This option extends option 1 with the ability to detect unknown licenses if multiple licenses are listed (usually through transitive dependencies that require the consumer's license to include their license). Examples:
      - If no license type can be detected at all (like in [rxdart 0.27.7](https://pub.dev/packages/rxdart/versions/0.27.7/license)), `Result.unclaimedTokenPercentage` is 1.
      - If the LICENSE file conforms to license format standards, `Result.unclaimedTokenPercentage` is somewhere between 0.01 and 0.05 (based on a test run with 40 dependencies).
      - If one standard license and one non-standard license is included, `Result.unclaimedTokenPercentage` will be around 0.5.
  3. We keep it as is. Non-standard licenses are ignored and not reported.
- A 100% confidence of reporting all licenses cannot be reached, independent of the chosen option.
- While libraries should ideally conform to recognized licensing standards, it is ultimately the consumer's responsibility to review and understand any non-standard licenses. If `very_good_license` ignores licenses, it may impose legal risks to the consumer.
- Proposal: Go with option 1. It handles most of the cases and the behaviour is comprehensible. What do you think, @alestiago? If you agree, I can implement it.

# Testing

A note on testing: PANA's `detectLicense` is currently mocked in the tests. I would recommend using the real implementation. Motivation: We rely on PANA's internal license detection algorithm, which is not exposed as a public API. If something in their implementation changes, we would neither notice through changelogs, nor our tests – unless we use the real implementation in our tests.
Hi @dominicmh thanks for the analysis!

I'll try having a read to the entire issue again next week and provide my two cents asap!

Fix patch:
diff --git a/lib/src/commands/packages/commands/check/commands/licenses.dart b/lib/src/commands/packages/commands/check/commands/licenses.dart
index 3d90c655a..8f345bbf6 100644
--- a/lib/src/commands/packages/commands/check/commands/licenses.dart
+++ b/lib/src/commands/packages/commands/check/commands/licenses.dart
@@ -286,7 +286,11 @@ class PackagesCheckLicensesCommand extends Command<int> {
           // ignore: invalid_use_of_visible_for_testing_member
           .map((match) => match.license.identifier)
           .toSet();
-      licenses[dependencyName] = rawLicense;
+      licenses[dependencyName] = {
+        ...rawLicense,
+        // If there are no matches, we add the unknown license
+        if (rawLicense.isEmpty) SpdxLicense.$unknown.value,
+      };
     }
 
     late final _BannedDependencyLicenseMap? bannedDependencies;


Test patch:
diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml
index f0ef1dba3..b3164e7c4 100644
--- a/.github/workflows/e2e.yaml
+++ b/.github/workflows/e2e.yaml
@@ -43,6 +43,7 @@ jobs:
           # E2E tests for the `packages check licenses` command
           - test/commands/packages/check/licenses/licenses_allowed_test.dart
           - test/commands/packages/check/licenses/licenses_forbidden_test.dart
+          - test/commands/packages/check/licenses/unknown_licenses_test.dart
 
     steps:
       - name: 📚 Git Checkout
diff --git a/e2e/helpers/command_helper.dart b/e2e/helpers/command_helper.dart
index f04e1d001..106134e37 100644
--- a/e2e/helpers/command_helper.dart
+++ b/e2e/helpers/command_helper.dart
@@ -32,6 +32,7 @@ void Function() withRunner(
     Logger logger,
     PubUpdater pubUpdater,
     List<String> printLogs,
+    List<String> progressLogs,
   )
   runnerFn,
 ) {
@@ -61,6 +62,12 @@ void Function() withRunner(
       () => pubUpdater.getLatestVersion(any()),
     ).thenAnswer((_) => Future.value('1.0.0'));
 
-    await runnerFn(commandRunner, logger, pubUpdater, printLogs);
+    await runnerFn(
+      commandRunner,
+      logger,
+      pubUpdater,
+      printLogs,
+      progressLogs,
+    );
   });
 }
diff --git a/e2e/test/commands/create/dart_cli/dart_cli_test.dart b/e2e/test/commands/create/dart_cli/dart_cli_test.dart
index 6732e9dbb..6de6ab40c 100644
--- a/e2e/test/commands/create/dart_cli/dart_cli_test.dart
+++ b/e2e/test/commands/create/dart_cli/dart_cli_test.dart
@@ -9,7 +9,7 @@ void main() {
   test(
     'create dart_cli',
     timeout: const Timeout(Duration(minutes: 2)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync();
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
diff --git a/e2e/test/commands/create/dart_package/dart_pkg_test.dart b/e2e/test/commands/create/dart_package/dart_pkg_test.dart
index a8b1d51fb..96d0c16f8 100644
--- a/e2e/test/commands/create/dart_package/dart_pkg_test.dart
+++ b/e2e/test/commands/create/dart_package/dart_pkg_test.dart
@@ -9,7 +9,7 @@ void main() {
   test(
     'create dart_package',
     timeout: const Timeout(Duration(minutes: 2)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync();
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
diff --git a/e2e/test/commands/create/docs_site/docs_site_test.dart b/e2e/test/commands/create/docs_site/docs_site_test.dart
index 632541a80..ed5fb4393 100644
--- a/e2e/test/commands/create/docs_site/docs_site_test.dart
+++ b/e2e/test/commands/create/docs_site/docs_site_test.dart
@@ -9,7 +9,7 @@ void main() {
   test(
     'create docs_site',
     timeout: const Timeout(Duration(minutes: 2)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync();
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
diff --git a/e2e/test/commands/create/flame_game/flame_game_test.dart b/e2e/test/commands/create/flame_game/flame_game_test.dart
index dd1dabc37..25710412e 100644
--- a/e2e/test/commands/create/flame_game/flame_game_test.dart
+++ b/e2e/test/commands/create/flame_game/flame_game_test.dart
@@ -9,7 +9,7 @@ void main() {
   test(
     'create flame_game',
     timeout: const Timeout(Duration(minutes: 5)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync();
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
diff --git a/e2e/test/commands/create/flutter_app/core_test.dart b/e2e/test/commands/create/flutter_app/core_test.dart
index 305e65b5c..d4d1e25d7 100644
--- a/e2e/test/commands/create/flutter_app/core_test.dart
+++ b/e2e/test/commands/create/flutter_app/core_test.dart
@@ -9,7 +9,7 @@ void main() {
   test(
     'create flutter_app',
     timeout: const Timeout(Duration(minutes: 2)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync();
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
diff --git a/e2e/test/commands/create/flutter_package/flutter_pkg_test.dart b/e2e/test/commands/create/flutter_package/flutter_pkg_test.dart
index 0abbd8004..0b9c8babc 100644
--- a/e2e/test/commands/create/flutter_package/flutter_pkg_test.dart
+++ b/e2e/test/commands/create/flutter_package/flutter_pkg_test.dart
@@ -9,7 +9,7 @@ void main() {
   test(
     'create flutter_package',
     timeout: const Timeout(Duration(minutes: 2)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync();
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
diff --git a/e2e/test/commands/create/flutter_plugin/flutter_plugin_test.dart b/e2e/test/commands/create/flutter_plugin/flutter_plugin_test.dart
index cee2fa3ef..66f0b417b 100644
--- a/e2e/test/commands/create/flutter_plugin/flutter_plugin_test.dart
+++ b/e2e/test/commands/create/flutter_plugin/flutter_plugin_test.dart
@@ -9,7 +9,7 @@ void main() {
   test(
     'create flutter_plugin',
     timeout: const Timeout(Duration(minutes: 8)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync();
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
diff --git a/e2e/test/commands/packages/check/licenses/licenses_allowed_test.dart b/e2e/test/commands/packages/check/licenses/licenses_allowed_test.dart
index 531ffcf9d..4a7834d60 100644
--- a/e2e/test/commands/packages/check/licenses/licenses_allowed_test.dart
+++ b/e2e/test/commands/packages/check/licenses/licenses_allowed_test.dart
@@ -15,7 +15,7 @@ void main() {
   test(
     'packages check licenses --allowed="MIT,BSD-3-Clause"',
     timeout: const Timeout(Duration(minutes: 2)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync();
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
@@ -52,6 +52,13 @@ void main() {
         equals(ExitCode.success.code),
         reason: 'Should succeed when allowed licenses are used',
       );
+
+      expect(
+        progressLogs,
+        contains(
+          '''Retrieved 2 licenses from 2 packages of type: MIT (1) and BSD-3-Clause (1).''',
+        ),
+      );
     }),
   );
 }
diff --git a/e2e/test/commands/packages/check/licenses/licenses_forbidden_test.dart b/e2e/test/commands/packages/check/licenses/licenses_forbidden_test.dart
index 8c8e78e2f..7cb4fc05d 100644
--- a/e2e/test/commands/packages/check/licenses/licenses_forbidden_test.dart
+++ b/e2e/test/commands/packages/check/licenses/licenses_forbidden_test.dart
@@ -1,6 +1,7 @@
 import 'dart:io';
 
 import 'package:mason/mason.dart';
+import 'package:mocktail/mocktail.dart';
 import 'package:path/path.dart' as path;
 import 'package:test/test.dart';
 
@@ -16,7 +17,7 @@ void main() {
   test(
     'packages check licenses --forbidden="MIT"',
     timeout: const Timeout(Duration(minutes: 2)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync();
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
@@ -54,6 +55,12 @@ void main() {
         equals(ExitCode.config.code),
         reason: 'Should fail when forbidden licenses are used',
       );
+
+      verify(
+        () => logger.err(
+          '1 dependency has a banned license: formz ([MIT](https://pub.dev/packages/formz/license)).',
+        ),
+      ).called(1);
     }),
   );
 }
diff --git a/e2e/test/commands/packages/check/licenses/unknown_licenses_test.dart b/e2e/test/commands/packages/check/licenses/unknown_licenses_test.dart
new file mode 100644
index 000000000..9431f6f32
--- /dev/null
+++ b/e2e/test/commands/packages/check/licenses/unknown_licenses_test.dart
@@ -0,0 +1,66 @@
+import 'dart:io';
+
+import 'package:mason/mason.dart';
+import 'package:path/path.dart' as path;
+import 'package:test/test.dart';
+
+import '../../../../../helpers/helpers.dart';
+
+/// Objectives:
+///
+/// * Generate a new Dart project using (`dart create`)
+/// * Add dependencies to `pubspec.yaml` with an unknown license
+/// * Run `very_good packages check licenses` and expect it to
+///   report 1 unknown license
+void main() {
+  test(
+    'packages check licenses (unknown license)',
+    timeout: const Timeout(Duration(minutes: 2)),
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
+      final tempDirectory = Directory.systemTemp.createTempSync();
+      addTearDown(() => tempDirectory.deleteSync(recursive: true));
+
+      const projectName = 'my_dart_project';
+      await expectSuccessfulProcessResult('dart', [
+        'create',
+        'my_dart_project',
+        '--no-pub',
+      ], workingDirectory: tempDirectory.path);
+      final projectPath = path.join(tempDirectory.path, projectName);
+      await expectSuccessfulProcessResult('dart', [
+        'pub',
+        'add',
+        'rxdart:0.27.7',
+      ], workingDirectory: projectPath);
+      await expectSuccessfulProcessResult('dart', [
+        'pub',
+        'get',
+      ], workingDirectory: projectPath);
+
+      final relativeProjectPath = path.relative(
+        projectPath,
+        from: Directory.current.path,
+      );
+
+      final result = await commandRunner.run([
+        'packages',
+        'check',
+        'licenses',
+        relativeProjectPath,
+      ]);
+
+      expect(
+        result,
+        equals(ExitCode.success.code),
+      );
+
+      expect(
+        progressLogs,
+        contains(
+          'Retrieved 2 licenses from 2 packages of type: BSD-3-Clause (1) '
+          'and unknown (1).',
+        ),
+      );
+    }),
+  );
+}
diff --git a/e2e/test/commands/test/async_main/async_main_test.dart b/e2e/test/commands/test/async_main/async_main_test.dart
index 69ab8553c..d26b6f5e8 100644
--- a/e2e/test/commands/test/async_main/async_main_test.dart
+++ b/e2e/test/commands/test/async_main/async_main_test.dart
@@ -9,7 +9,7 @@ void main() {
   test(
     'supports async main methods',
     timeout: const Timeout(Duration(minutes: 2)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync('async_main');
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
diff --git a/e2e/test/commands/test/compilation_error/compilation_error_test.dart b/e2e/test/commands/test/compilation_error/compilation_error_test.dart
index 2dd4777c6..1190104bf 100644
--- a/e2e/test/commands/test/compilation_error/compilation_error_test.dart
+++ b/e2e/test/commands/test/compilation_error/compilation_error_test.dart
@@ -9,7 +9,7 @@ void main() {
   test(
     'fails when there is a compilation error, but does not crash',
     timeout: const Timeout(Duration(minutes: 2)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync(
         'compilation_error',
       );
diff --git a/e2e/test/commands/test/no_project/no_project_test.dart b/e2e/test/commands/test/no_project/no_project_test.dart
index bbc5f547d..c30620e38 100644
--- a/e2e/test/commands/test/no_project/no_project_test.dart
+++ b/e2e/test/commands/test/no_project/no_project_test.dart
@@ -8,7 +8,7 @@ import '../../../../helpers/helpers.dart';
 void main() {
   test(
     'fails if the project does not exist',
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync('no_project');
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
diff --git a/e2e/test/commands/test/spaced_golden_file_name/spaced_golden_file_name_test.dart b/e2e/test/commands/test/spaced_golden_file_name/spaced_golden_file_name_test.dart
index 6910825cf..48de65f4b 100644
--- a/e2e/test/commands/test/spaced_golden_file_name/spaced_golden_file_name_test.dart
+++ b/e2e/test/commands/test/spaced_golden_file_name/spaced_golden_file_name_test.dart
@@ -9,7 +9,7 @@ void main() {
   test(
     'allows golden files with spaces in the name',
     timeout: const Timeout(Duration(minutes: 2)),
-    withRunner((commandRunner, logger, updater, logs) async {
+    withRunner((commandRunner, logger, updater, logs, progressLogs) async {
       final tempDirectory = Directory.systemTemp.createTempSync('async_main');
       addTearDown(() => tempDirectory.deleteSync(recursive: true));
 
diff --git a/test/src/commands/packages/commands/check/commands/licenses_test.dart b/test/src/commands/packages/commands/check/commands/licenses_test.dart
index 543dbe128..ee610704c 100644
--- a/test/src/commands/packages/commands/check/commands/licenses_test.dart
+++ b/test/src/commands/packages/commands/check/commands/licenses_test.dart
@@ -319,6 +319,97 @@ void main() {
             expect(result, equals(ExitCode.success.code));
           }),
         );
+
+        test(
+          'unknown when non-standard license file is found',
+          withRunner((commandRunner, logger, pubUpdater, printLogs) async {
+            File(
+              path.join(tempDirectory.path, pubspecLockBasename),
+            ).writeAsStringSync(_validPubspecLockContent);
+
+            when(
+              () => packageConfig.packages,
+            ).thenReturn([veryGoodTestRunnerConfigPackage]);
+            final licenseFilePath = path.join(
+              tempDirectory.path,
+              veryGoodTestRunnerConfigPackage.name,
+              'LICENSE',
+            );
+            File(licenseFilePath).writeAsStringSync('''
+Licensed under the Apache License, Version 2.0 (the "License"); you
+may not use this file except in compliance with the License. You may
+obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+implied. See the License for the specific language governing permissions
+and limitations under the License.''');
+
+            when(() => logger.progress(any())).thenReturn(progress);
+            when(() => detectorResult.matches).thenReturn([]);
+
+            final result = await commandRunner.run(
+              [...commandArguments, tempDirectory.path],
+            );
+
+            verify(
+              () => progress.update(
+                'Collecting licenses from 1 out of 1 package',
+              ),
+            ).called(1);
+            verify(
+              () => progress.complete(
+                '''Retrieved 1 license from 1 package of type: unknown (1).''',
+              ),
+            ).called(1);
+
+            expect(result, equals(ExitCode.success.code));
+          }),
+        );
+
+        test(
+          'unknown when invalid license file is found',
+          withRunner((commandRunner, logger, pubUpdater, printLogs) async {
+            File(
+              path.join(tempDirectory.path, pubspecLockBasename),
+            ).writeAsStringSync(_validPubspecLockContent);
+
+            when(
+              () => packageConfig.packages,
+            ).thenReturn([veryGoodTestRunnerConfigPackage]);
+            final licenseFilePath = path.join(
+              tempDirectory.path,
+              veryGoodTestRunnerConfigPackage.name,
+              'LICENSE',
+            );
+            File(
+              licenseFilePath,
+            ).writeAsStringSync('This is an invalid license file.');
+
+            when(() => logger.progress(any())).thenReturn(progress);
+            when(() => detectorResult.matches).thenReturn([]);
+
+            final result = await commandRunner.run(
+              [...commandArguments, tempDirectory.path],
+            );
+
+            verify(
+              () => progress.update(
+                'Collecting licenses from 1 out of 1 package',
+              ),
+            ).called(1);
+            verify(
+              () => progress.complete(
+                '''Retrieved 1 license from 1 package of type: unknown (1).''',
+              ),
+            ).called(1);
+
+            expect(result, equals(ExitCode.success.code));
+          }),
+        );
       },
     );