Repository: mochajs/mocha

Problem statement:
🚀 Feature: Possibility to return 0 exit code when tests run successfully even with fails
**Is your feature request related to a problem or a nice-to-have?? Please describe.**
A clear and concise description of what the problem is. E.g. I'm always frustrated when [...]

Sometimes it is useful to separate situations when tests run successfully, and there is no infrastructure or runtime problems, and when there are some runtime problems.
Possibility to return non-zero exit code only on infrastructure or runtime problems make easier integration with different toolchain.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

Flag, something like `failOnFailingTests` like in Karma (https://github.com/karma-runner/karma/blob/master/docs/config/01-configuration-file.md#failonfailingtestsuite).
- 0 exit code when tests run successfully
- 1 exit code when something went wrong

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

Karma 
https://github.com/karma-runner/karma/blob/master/docs/config/01-configuration-file.md#failonfailingtestsuite

**Additional context**
Add any other context or screenshots about the feature request here.



Hints:
> Sometimes it is useful to separate situations when tests run successfully, and there is no infrastructure or runtime problems, and when there are some runtime problems.

I didn't understand it. Could you explain more detail for usecases.
Using Mocha in some build tools (for example Gradle).
After Mocha successfully run, I know test status via custom loader. Then I provide test results to "parent" build system and doesn't matter on Mocha exit code. Further results will be processed, and I don't want to fail "parent" build system.
But it does matter if there is some problems with infrastructure (for example Node problem). And in this case I want to get non-zero exit code of Mocha and force to fail "parent" build system.

But for now I have to fail of "parent" build system on both cases, when tests are failed which is **valid** situation for "parent" build system, and infrastructure problem (syntax problem e.g.) which is **invalid** situation for "parent" build system.

If I ignore exit code of Mocha, there is no fail on failed tests (and it is OK) and on infrastructure problem (and it is not OK)
Thanks.  This is a duplicate of...several issues.  Can you run `mocha <args> || true` instead?  This is how I've seen people treat any result as "passing"

@boneskull Thank you for reply.
I don't need any result as "passing", I think that in topic starter I incorrectly expressed the thought.
Let's imagine we have states:
- Tests passed and run successful - 0 failed tests and 0 zero code
- Some tests failed but run **successful** - some failed tests and 0 zero code
- Run **failed** - some infrastructure problems (for example don't find some critical module)

I want to get zero code only if no "infrastructure" problem. If I run `mocha <args> || true` I get zero code even in this case.

It is similar with next property in Karma
https://github.com/karma-runner/karma/blob/master/docs/config/01-configuration-file.md#failonfailingtestsuite



Patch:
diff --git a/lib/cli/run-helpers.js b/lib/cli/run-helpers.js
--- a/lib/cli/run-helpers.js
+++ b/lib/cli/run-helpers.js
@@ -21,25 +21,24 @@ const {UnmatchedFile} = require('./collect-files');
 
 /**
  * Exits Mocha when tests + code under test has finished execution (default)
- * @param {number} code - Exit code; typically # of failures
+ * @param {number} clampedCode - Exit code; typically # of failures
  * @ignore
  * @private
  */
-const exitMochaLater = code => {
+const exitMochaLater = clampedCode => {
   process.on('exit', () => {
-    process.exitCode = Math.min(code, 255);
+    process.exitCode = clampedCode;
   });
 };
 
 /**
  * Exits Mocha when Mocha itself has finished execution, regardless of
  * what the tests or code under test is doing.
- * @param {number} code - Exit code; typically # of failures
+ * @param {number} clampedCode - Exit code; typically # of failures
  * @ignore
  * @private
  */
-const exitMocha = code => {
-  const clampedCode = Math.min(code, 255);
+const exitMocha = clampedCode => {
   let draining = 0;
 
   // Eagerly set the process's exit code in case stream.write doesn't
@@ -139,12 +138,17 @@ const handleUnmatchedFiles = (mocha, unmatchedFiles) => {
  * @param {Mocha} mocha - Mocha instance
  * @param {Options} [opts] - Command line options
  * @param {boolean} [opts.exit] - Whether or not to force-exit after tests are complete
+ * @param {boolean} [opts.passOnFailingTestSuite] - Whether or not to fail test run if tests were failed
  * @param {Object} fileCollectParams - Parameters that control test
  *   file collection. See `lib/cli/collect-files.js`.
  * @returns {Promise<Runner>}
  * @private
  */
-const singleRun = async (mocha, {exit}, fileCollectParams) => {
+const singleRun = async (
+  mocha,
+  {exit, passOnFailingTestSuite},
+  fileCollectParams
+) => {
   const fileCollectionObj = collectFiles(fileCollectParams);
 
   if (fileCollectionObj.unmatchedFiles.length > 0) {
@@ -156,7 +160,9 @@ const singleRun = async (mocha, {exit}, fileCollectParams) => {
 
   // handles ESM modules
   await mocha.loadFilesAsync();
-  return mocha.run(exit ? exitMocha : exitMochaLater);
+  return mocha.run(
+    createExitHandler({exit, passOnFailingTestSuite})
+  );
 };
 
 /**
@@ -186,7 +192,9 @@ const parallelRun = async (mocha, options, fileCollectParams) => {
   mocha.files = fileCollectionObj.files;
 
   // note that we DO NOT load any files here; this is handled by the worker
-  return mocha.run(options.exit ? exitMocha : exitMochaLater);
+  return mocha.run(
+    createExitHandler(options)
+  );
 };
 
 /**
@@ -282,3 +290,15 @@ exports.validateLegacyPlugin = (opts, pluginType, map = {}) => {
     }
   }
 };
+
+const createExitHandler = ({ exit, passOnFailingTestSuite }) => {
+  return code => {
+    const clampedCode = passOnFailingTestSuite
+      ? 0
+      : Math.min(code, 255);
+
+    return exit
+      ? exitMocha(clampedCode)
+      : exitMochaLater(clampedCode);
+  };
+};
diff --git a/lib/cli/run-option-metadata.js b/lib/cli/run-option-metadata.js
--- a/lib/cli/run-option-metadata.js
+++ b/lib/cli/run-option-metadata.js
@@ -35,6 +35,7 @@ const TYPES = (exports.types = {
     'diff',
     'dry-run',
     'exit',
+    'pass-on-failing-test-suite',
     'fail-zero',
     'forbid-only',
     'forbid-pending',
diff --git a/lib/cli/run.js b/lib/cli/run.js
--- a/lib/cli/run.js
+++ b/lib/cli/run.js
@@ -98,6 +98,11 @@ exports.builder = yargs =>
         requiresArg: true,
         coerce: list
       },
+      'pass-on-failing-test-suite': {
+        default: false,
+        description: 'Not fail test run if tests were failed',
+        group: GROUPS.RULES
+      },
       'fail-zero': {
         description: 'Fail test run if no test(s) encountered',
         group: GROUPS.RULES
diff --git a/lib/mocha.js b/lib/mocha.js
--- a/lib/mocha.js
+++ b/lib/mocha.js
@@ -157,6 +157,7 @@ exports.run = function (...args) {
  * @param {boolean} [options.delay] - Delay root suite execution?
  * @param {boolean} [options.diff] - Show diff on failure?
  * @param {boolean} [options.dryRun] - Report tests without running them?
+ * @param {boolean} [options.passOnFailingTestSuite] - Fail test run if tests were failed?
  * @param {boolean} [options.failZero] - Fail test run if zero tests?
  * @param {string} [options.fgrep] - Test filter given string.
  * @param {boolean} [options.forbidOnly] - Tests marked `only` fail the suite?
@@ -216,6 +217,7 @@ function Mocha(options = {}) {
     'delay',
     'diff',
     'dryRun',
+    'passOnFailingTestSuite',
     'failZero',
     'forbidOnly',
     'forbidPending',
@@ -870,6 +872,20 @@ Mocha.prototype.failZero = function (failZero) {
   return this;
 };
 
+/**
+ * Fail test run if tests were failed.
+ *
+ * @public
+ * @see [CLI option](../#-pass-on-failing-test-suite)
+ * @param {boolean} [passOnFailingTestSuite=false] - Whether to fail test run.
+ * @return {Mocha} this
+ * @chainable
+ */
+Mocha.prototype.passOnFailingTestSuite = function(passOnFailingTestSuite) {
+  this.options.passOnFailingTestSuite = passOnFailingTestSuite === true;
+  return this;
+};
+
 /**
  * Causes tests marked `only` to fail the suite.
  *


Test patch:
diff --git a/test/integration/fixtures/failing-sync.fixture.js b/test/integration/fixtures/failing-sync.fixture.js
new file mode 100644
--- /dev/null
+++ b/test/integration/fixtures/failing-sync.fixture.js
@@ -0,0 +1,9 @@
+'use strict';
+
+var assert = require('assert');
+
+describe('a suite', function() {
+  it('should succeed', function() {
+    assert(false);
+  });
+});
diff --git a/test/integration/options/passOnFailingTestSuite.spec.js b/test/integration/options/passOnFailingTestSuite.spec.js
new file mode 100644
--- /dev/null
+++ b/test/integration/options/passOnFailingTestSuite.spec.js
@@ -0,0 +1,40 @@
+'use strict';
+
+var helpers = require('../helpers');
+var runMochaJSON = helpers.runMochaJSON;
+
+describe('Enabled --pass-on-failing-test-suite', function() {
+  var args = ['--pass-on-failing-test-suite=true'];
+
+  it('Test should finish with zero code with disabled option', function(done) {
+    var fixture = 'failing-sync.fixture.js';
+    runMochaJSON(fixture, args, function(err, res) {
+      if (err) {
+        return done(err);
+      }
+
+      expect(res, 'to have passed test count', 0)
+        .and('to have test count', 1)
+        .and('to have exit code', 0);
+      done();
+    });
+  });
+});
+
+describe('Disabled --pass-on-failing-test-suite', function() {
+  var args = ['--pass-on-failing-test-suite=false'];
+
+  it('Test should return non-zero code with enabled option', function(done) {
+    var fixture = 'failing-sync.fixture.js';
+    runMochaJSON(fixture, args, function(err, res) {
+      if (err) {
+        return done(err);
+      }
+
+      expect(res, 'to have passed test count', 0)
+        .and('to have test count', 1)
+        .and('to have exit code', 1);
+      done();
+    });
+  });
+});
diff --git a/test/unit/mocha.spec.js b/test/unit/mocha.spec.js
--- a/test/unit/mocha.spec.js
+++ b/test/unit/mocha.spec.js
@@ -376,6 +376,28 @@ describe('Mocha', function () {
       });
     });
 
+    describe('passOnFailingTestSuite()', function() {
+      it('should set the passOnFailingTestSuite option to false', function() {
+        mocha.passOnFailingTestSuite();
+        expect(
+          mocha.options,
+          'to have property',
+          'passOnFailingTestSuite',
+          false
+        );
+      });
+
+      it('should set the passOnFailingTestSuite option to true', function() {
+        mocha.passOnFailingTestSuite(true);
+        expect(
+          mocha.options,
+          'to have property',
+          'passOnFailingTestSuite',
+          true
+        );
+      });
+    });
+
     describe('failZero()', function () {
       it('should set the failZero option to true', function () {
         mocha.failZero();