Repository: dotnet/BenchmarkDotNet

Issue:
Can't use latest version of BenchmarkDotNet

I tried to use the latest bdn master (dotnet/BenchmarkDotNet@73f8fd1dc49f34b79afa4ef16e0204b1369fdd1a) here, but trying to run benchmarks fails with:

```
Unhandled exception. System.InvalidOperationException: Failed to compare two elements in the array.
 ---> System.ArgumentException: At least one object must implement IComparable.
   at System.Collections.Comparer.Compare(Object a, Object b)
   at System.ValueTuple`3.CompareTo(ValueTuple`3 other)
   at System.ValueTuple`3.System.IComparable.CompareTo(Object other)
   at BenchmarkDotNet.Parameters.ParameterComparer.CompareValues(Object x, Object y)
   at BenchmarkDotNet.Parameters.ParameterComparer.Compare(ParameterInstances x, ParameterInstances y)
   at BenchmarkDotNet.Order.DefaultOrderer.BenchmarkComparer.Compare(BenchmarkCase x, BenchmarkCase y)
   at System.Collections.Generic.ArraySortHelper`1.SwapIfGreater(Span`1 keys, Comparison`1 comparer, Int32 i, Int32 j)
   at System.Collections.Generic.ArraySortHelper`1.PickPivotAndPartition(Span`1 keys, Comparison`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.IntroSort(Span`1 keys, Int32 depthLimit, Comparison`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.IntrospectiveSort(Span`1 keys, Comparison`1 comparer)
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(Span`1 keys, IComparer`1 comparer)
   --- End of inner exception stack trace ---
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(Span`1 keys, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
   at BenchmarkDotNet.Order.DefaultOrderer.GetExecutionOrder(ImmutableArray`1 benchmarkCases, IEnumerable`1 order)
   at BenchmarkDotNet.Running.BenchmarkConverter.MethodsToBenchmarksWithFullConfig(Type type, MethodInfo[] benchmarkMethods, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkConverter.TypeToBenchmarks(Type type, IConfig config)
   at BenchmarkDotNet.Running.TypeFilter.<>c__DisplayClass1_0.<Filter>b__0(Type type)
   at System.Linq.Enumerable.SelectListIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.ToArray()
   at BenchmarkDotNet.Running.TypeFilter.Filter(IConfig effectiveConfig, IEnumerable`1 types)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config, Boolean askUserForInput)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
   at MicroBenchmarks.Program.Main(String[] args) in /Users/ankj/dev/performance/src/benchmarks/micro/Program.cs:line 43
```

This seems to be failing for
```
System.ValueTuple`3[System.Globalization.CultureInfo,System.Globalization.CompareOptions,System.Boolean], (en-US, Ordinal, False)
System.ValueTuple`3[System.Globalization.CultureInfo,System.Globalization.CompareOptions,System.Boolean], (pl-PL, None, False)`
```

.. , because `CultureInfo` does not implement `IComparable`.

IIUC, this is a new code path introduced as part of https://github.com/dotnet/BenchmarkDotNet/pull/2304 .

And it fails for https://github.com/dotnet/performance/blob/dd14c7d44444a0211035af8464b743a55b4dd55e/src/benchmarks/micro/libraries/System.Globalization/StringSearch.cs#L46 .

cc @mrahhal @adamsitnik @LoopedBard3 


Hints:
Should this be in dotnet/BenchmarkDotNet instead?
The following reproduces the problem:

```cs
var t1 = (new CultureInfo("en-US"), CompareOptions.Ordinal, false);
var t2 = (new CultureInfo("pl-PL"), CompareOptions.None, false);

Assert.IsTrue(typeof(IComparable).IsAssignableFrom(t1.GetType())); // Succeeds

var result = t1.CompareTo(t2); // Throws System.ArgumentException: At least one object must implement IComparable.
```

This is because `ValueTuple` implements `IComparable` regardless of whether or not the inner types inside imeplement it, so its `CompareTo` implementation is fallible and might throw in that case.

In this particular case, `CultureInfo` doesn't, and so it throws.

One potential solution is to have specific support for detecting `ValueTuple`s, and do an alternative check of whether all the types inside are `IComparable` to consider the whole tuple `IComparable`. Or the easy way out, if it's a `ValueTuple` then just skip the `IComparable` support and go straight to the `ToString()` comparison.

For reference, the `ValueTuple`'s `CompareTo` [implementation](https://source.dot.net/#System.Private.CoreLib/src/libraries/System.Private.CoreLib/src/System/ValueTuple.cs,722), and [this is](https://github.com/mrahhal/BenchmarkDotNet/blob/ca4d012c3b9c1a8bac230b8311ea2f7db1d13d55/src/BenchmarkDotNet/Parameters/ParameterComparer.cs#L28-L32) where we might solve this in BenchmarkDotNet.
Working on this fix. Let me know which of the following two solutions I listed above do you prefer:

1. For `ValueTuple`s, check all inner types if they implement `IComparable` before considering it such
2. For `ValueTuple`s, just skip `IComparable` support and go straight to string comparison (emulates old behavior for `ValueTuple`s)

Any other suggested solution?
@mrahhal Maybe just catch the exception and fall back to string comparison?
Could do that, but that might hide other unrelated exceptions we didn't intend to hide. Might be better to properly handle this particular case in my opinion. `ValueTuple` seems to be a special case, `IComparable` doesn't usually throw.
@mrahhal How is the fix looking? Any ETA? Thanks.
Was waiting for a reply as I didn't want to work on something and then change it right after a review. And honestly forgot about it while waiting. Ready to roll out a fix when a maintainer confirms one of my two suggestions above..

Or I think I'll just go with whatever I think is better to start with. Will submit a PR soon.
I think if you want to handle `ValueTuple` specially, it's fine.

> check all inner types if they implement `IComparable` before considering it such

Just be aware that they can have variable number of types and nested `ValueTuple`s.
> Just be aware that they can have variable number of types and nested ValueTuples.

Going through `ITuple` might make that easier.
> Going through `ITuple` might make that easier.

That's an internal interface (and it only has a Size prop, I don't think it helps much anyway).

> Just be aware that they can have variable number of types and nested `ValueTuple`s.

Yes. A simple recursive check will do for nesting. The problem is actually how to detect `ValueTuple`s, since it's a series of (unlimited number of?) structs depending on the number of items. And I don't feel either supporting up to some fixed number or doing something fancy with `Type.FullName` is a good idea.

Starting to think handling exceptions when calling the `IComparable` impl and falling back to `.ToString()` as @timcassell suggested is just the easiest as a start to get the old behavior back. Mayhe this can be handled better in the future, but let's just roll out a fix for now.
> That's an internal interface (and it only has a Size prop, I don't think it helps much anyway).

It's public interface. And has also indexer.
> It's public interface. And has also indexer.

I think it's internal in netstandard. But more importantly, isn't `ITuple` for `Tuple` (the older class tuples)? I don't think it's related to `ValueTuple`.
Yeah, you need at least `netstandard2.1`. :(

Both `Tuple` and `ValueTuple`. 
BDN has a transitive reference to the `System.ValueTuple` nuget package, so you can use the `ITuple` interface. In fact, you should use that interface to check for `Tuple` also, as it also has the same `IComparable` functionality.

```cs
public interface ITuple
{
    int Length { get; }
    object this[int index] { get; }
}
```
![image](https://github.com/dotnet/BenchmarkDotNet/assets/4404199/bbb42ca8-32bf-47bd-8b11-8795fedd4d59)

Doesn't seem usable in netstandard2.0.

And yes, `Tuple` has the same problem. Looking at the `ITuple` interface above, indeed it would have been the best way to solve this, but I guess we're stuck because of netstandard2.0.
Oh weird, it's not public from the nuget package. You may still be able to use it with reflection, though.

Patch:
diff --git a/src/BenchmarkDotNet/Parameters/ParameterComparer.cs b/src/BenchmarkDotNet/Parameters/ParameterComparer.cs
index a1363de073..6cadf6a946 100644
--- a/src/BenchmarkDotNet/Parameters/ParameterComparer.cs
+++ b/src/BenchmarkDotNet/Parameters/ParameterComparer.cs
@@ -28,7 +28,16 @@ private int CompareValues(object x, object y)
             if (x != null && y != null && x.GetType() == y.GetType() &&
                 x is IComparable xComparable)
             {
-                return xComparable.CompareTo(y);
+                try
+                {
+                    return xComparable.CompareTo(y);
+                }
+                // Some types, such as Tuple and ValueTuple, have a fallible CompareTo implementation which can throw if the inner items don't implement IComparable.
+                // See: https://github.com/dotnet/BenchmarkDotNet/issues/2346
+                // For now, catch and ignore the exception, and fallback to string comparison below.
+                catch (ArgumentException ex) when (ex.Message.Contains("At least one object must implement IComparable."))
+                {
+                }
             }
 
             // Anything else.


Test patch:
diff --git a/tests/BenchmarkDotNet.Tests/ParameterComparerTests.cs b/tests/BenchmarkDotNet.Tests/ParameterComparerTests.cs
index 0f01045b5c..2ef1bc8891 100644
--- a/tests/BenchmarkDotNet.Tests/ParameterComparerTests.cs
+++ b/tests/BenchmarkDotNet.Tests/ParameterComparerTests.cs
@@ -157,6 +157,70 @@ public void IComparableComparisionTest()
             Assert.Equal(4, ((ComplexParameter)sortedData[3].Items[0].Value).Value);
         }
 
+        [Fact]
+        public void ValueTupleWithNonIComparableInnerTypesComparisionTest()
+        {
+            var comparer = ParameterComparer.Instance;
+            var originalData = new[]
+            {
+                new ParameterInstances(new[]
+                {
+                    new ParameterInstance(sharedDefinition, (new ComplexNonIComparableParameter(), 1), null)
+                }),
+                new ParameterInstances(new[]
+                {
+                    new ParameterInstance(sharedDefinition, (new ComplexNonIComparableParameter(), 3), null)
+                }),
+                new ParameterInstances(new[]
+                {
+                    new ParameterInstance(sharedDefinition, (new ComplexNonIComparableParameter(), 2), null)
+                }),
+                new ParameterInstances(new[]
+                {
+                    new ParameterInstance(sharedDefinition, (new ComplexNonIComparableParameter(), 4), null)
+                })
+            };
+
+            var sortedData = originalData.OrderBy(d => d, comparer).ToArray();
+
+            Assert.Equal(1, (((ComplexNonIComparableParameter, int))sortedData[0].Items[0].Value).Item2);
+            Assert.Equal(2, (((ComplexNonIComparableParameter, int))sortedData[1].Items[0].Value).Item2);
+            Assert.Equal(3, (((ComplexNonIComparableParameter, int))sortedData[2].Items[0].Value).Item2);
+            Assert.Equal(4, (((ComplexNonIComparableParameter, int))sortedData[3].Items[0].Value).Item2);
+        }
+
+        [Fact]
+        public void TupleWithNonIComparableInnerTypesComparisionTest()
+        {
+            var comparer = ParameterComparer.Instance;
+            var originalData = new[]
+            {
+                new ParameterInstances(new[]
+                {
+                    new ParameterInstance(sharedDefinition, Tuple.Create(new ComplexNonIComparableParameter(), 1), null)
+                }),
+                new ParameterInstances(new[]
+                {
+                    new ParameterInstance(sharedDefinition, Tuple.Create(new ComplexNonIComparableParameter(), 3), null)
+                }),
+                new ParameterInstances(new[]
+                {
+                    new ParameterInstance(sharedDefinition, Tuple.Create(new ComplexNonIComparableParameter(), 2), null)
+                }),
+                new ParameterInstances(new[]
+                {
+                    new ParameterInstance(sharedDefinition, Tuple.Create(new ComplexNonIComparableParameter(), 4), null)
+                })
+            };
+
+            var sortedData = originalData.OrderBy(d => d, comparer).ToArray();
+
+            Assert.Equal(1, ((Tuple<ComplexNonIComparableParameter, int>)sortedData[0].Items[0].Value).Item2);
+            Assert.Equal(2, ((Tuple<ComplexNonIComparableParameter, int>)sortedData[1].Items[0].Value).Item2);
+            Assert.Equal(3, ((Tuple<ComplexNonIComparableParameter, int>)sortedData[2].Items[0].Value).Item2);
+            Assert.Equal(4, ((Tuple<ComplexNonIComparableParameter, int>)sortedData[3].Items[0].Value).Item2);
+        }
+
         private class ComplexParameter : IComparable<ComplexParameter>, IComparable
         {
             public ComplexParameter(int value, string name)
@@ -199,5 +263,9 @@ public int CompareTo(object obj)
                 return CompareTo(other);
             }
         }
+
+        private class ComplexNonIComparableParameter
+        {
+        }
     }
 }
\ No newline at end of file