Repository: restsharp/RestSharp

Issue:
DownloadStreamAsync does not handle 404 pages?
**DO NOT USE ISSUES FOR QUESTIONS**

**Describe the bug**
As it stands currently, DownloadStreamAsync executes the request and blindly downloads whatever response it gets back regardless of whether the request was successful on or. So if you have someone rendering a fancy 404 page like this one:

http://bti-usa.com/images/pictures/sf/sf3058.jpg

It gets happily downloaded as a file (which of course is not actually a jpg file, but that is not the root issue here :) ).

**To Reproduce**
Download the URL above using DownloadStreamAsync and it will download the contents.

**Expected behavior**
I think the correct behavior here is that DownloadStreamAsync should check for a valid HTTP status (say 200/OK?) and then return null or something if the status is not valid?

I am not sure if that would be a breaking change or not, as it is not clear what the callers would expect for the resulting stream on error. Perhaps it would expect an exception to be thrown if something was wrong, otherwise it expects a valid stream and would not expect a null (and would crash with an NRE). 

For now I am simply going to implement my own download code that handles the response codes, but I think this is something that the library should be responsible for doing correctly?


Hints:
Yeah, that's a miss. I will add a test, and then we can decide what to do with it. If the client is set to throw, I can accommodate that. However, 404 isn't currently considered as an error. Maybe returning nothing would be a good solution?
I already have a fix in my own branch, so you could pull that over for the 404 handling? Also added an exception when we attempt to use a disposed RestClient :)

https://github.com/kendallb/RestSharp/commit/27c47663a0ea9bf10206125d49fdf00d1a83d10f
I think adding this would be too big of a change, as it doesn't work this way now

```
        if (!responseMessage.IsSuccessStatusCode) {
            var statusCode = responseMessage.StatusCode;
            throw new WebException($"The remote server returned an error: ({(int)statusCode}) {statusCode}");
        }
```

What I can do is to move the check downstream, particularly to the `Download` function. I am not sure yet if it's doable.

The default behaviour was always that RestSharp doesn't throw by default and rather return an error response. For example, you can examine the response content to find out what it says. The client throws in `GetAsync` etc overloads, it's by design, or when the client `ThrowOnAnyError` is set. As the download function is relatively new, I can change its behaviour without breaking too much, but it still needs to be a major version update. 
Well I am not sure you can move it any higher up, as the function I am calling is DownloadStreamAsync so for my use case it is already as high up the food chain as possible.

So while it is not how it functions today, it is also impossible to implement the mechanism of returning an Exception from that function because there is no return response from DownloadStream or Download to inspect. We could change it to only throw that exception if Options.ThrowOnAnyError is set so it would be the same as the exception handling above it, but I do wonder if that option makes any sense at all inside DownloadStreamAsync because there is no way for the caller to ever get the underlying exception that occurred if Options.ThrowOnAnyError is false? It will just silently fail and you get really strange results downloaded.



Patch:
diff --git a/src/RestSharp/Extensions/HttpResponseExtensions.cs b/src/RestSharp/Extensions/HttpResponseExtensions.cs
new file mode 100644
index 000000000..53b873e6c
--- /dev/null
+++ b/src/RestSharp/Extensions/HttpResponseExtensions.cs
@@ -0,0 +1,27 @@
+//  Copyright (c) .NET Foundation and Contributors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+namespace RestSharp.Extensions;
+
+public static class HttpResponseExtensions {
+    internal static Exception? MaybeException(this HttpResponseMessage httpResponse)
+        => httpResponse.IsSuccessStatusCode
+            ? null
+#if NETSTANDARD
+            : new HttpRequestException($"Request failed with status code {httpResponse.StatusCode}");
+#else
+            : new HttpRequestException($"Request failed with status code {httpResponse.StatusCode}", null, httpResponse.StatusCode);
+#endif
+}
diff --git a/src/RestSharp/Response/RestResponse.cs b/src/RestSharp/Response/RestResponse.cs
index 503d2c7f2..a9225ec23 100644
--- a/src/RestSharp/Response/RestResponse.cs
+++ b/src/RestSharp/Response/RestResponse.cs
@@ -89,7 +89,7 @@ async Task<RestResponse> GetDefaultResponse() {
                 ContentLength       = httpResponse.Content.Headers.ContentLength,
                 ContentType         = httpResponse.Content.Headers.ContentType?.MediaType,
                 ResponseStatus      = calculateResponseStatus(httpResponse),
-                ErrorException      = MaybeException(),
+                ErrorException      = httpResponse.MaybeException(),
                 ResponseUri         = httpResponse.RequestMessage!.RequestUri,
                 Server              = httpResponse.Headers.Server.ToString(),
                 StatusCode          = httpResponse.StatusCode,
@@ -102,15 +102,6 @@ async Task<RestResponse> GetDefaultResponse() {
                 RootElement         = request.RootElement
             };
 
-            Exception? MaybeException()
-                => httpResponse.IsSuccessStatusCode
-                    ? null
-#if NETSTANDARD
-                    : new HttpRequestException($"Request failed with status code {httpResponse.StatusCode}");
-#else
-                    : new HttpRequestException($"Request failed with status code {httpResponse.StatusCode}", null, httpResponse.StatusCode);
-#endif
-
             Task<Stream?> ReadResponse() => httpResponse.ReadResponse(cancellationToken);
 
             async Task<Stream?> ReadAndConvertResponse() {
diff --git a/src/RestSharp/RestClient.Async.cs b/src/RestSharp/RestClient.Async.cs
index 7a353e550..0a57d189f 100644
--- a/src/RestSharp/RestClient.Async.cs
+++ b/src/RestSharp/RestClient.Async.cs
@@ -50,8 +50,7 @@ async Task<InternalResponse> ExecuteInternal(RestRequest request, CancellationTo
 
         using var requestContent = new RequestContent(this, request);
 
-        if (Authenticator != null)
-            await Authenticator.Authenticate(this, request).ConfigureAwait(false);
+        if (Authenticator != null) await Authenticator.Authenticate(this, request).ConfigureAwait(false);
 
         var httpMethod = AsHttpMethod(request.Method);
         var url        = BuildUri(request);
@@ -61,7 +60,8 @@ async Task<InternalResponse> ExecuteInternal(RestRequest request, CancellationTo
 
         using var timeoutCts = new CancellationTokenSource(request.Timeout > 0 ? request.Timeout : int.MaxValue);
         using var cts        = CancellationTokenSource.CreateLinkedTokenSource(timeoutCts.Token, cancellationToken);
-        var       ct         = cts.Token;
+
+        var ct = cts.Token;
 
         try {
             var headers = new RequestHeaders()
@@ -70,13 +70,11 @@ async Task<InternalResponse> ExecuteInternal(RestRequest request, CancellationTo
                 .AddAcceptHeader(AcceptedContentTypes);
             message.AddHeaders(headers);
 
-            if (request.OnBeforeRequest != null)
-                await request.OnBeforeRequest(message).ConfigureAwait(false);
+            if (request.OnBeforeRequest != null) await request.OnBeforeRequest(message).ConfigureAwait(false);
 
             var responseMessage = await HttpClient.SendAsync(message, request.CompletionOption, ct).ConfigureAwait(false);
 
-            if (request.OnAfterRequest != null)
-                await request.OnAfterRequest(responseMessage).ConfigureAwait(false);
+            if (request.OnAfterRequest != null) await request.OnAfterRequest(responseMessage).ConfigureAwait(false);
 
             return new InternalResponse(responseMessage, url, null, timeoutCts.Token);
         }
@@ -99,8 +97,10 @@ record InternalResponse(HttpResponseMessage? ResponseMessage, Uri Url, Exception
         request.CompletionOption = HttpCompletionOption.ResponseHeadersRead;
         var response = await ExecuteInternal(request, cancellationToken).ConfigureAwait(false);
 
-        if (response.Exception != null) {
-            return Options.ThrowOnAnyError ? throw response.Exception : null;
+        var exception = response.Exception ?? response.ResponseMessage?.MaybeException();
+
+        if (exception != null) {
+            return Options.ThrowOnAnyError ? throw exception : null;
         }
 
         if (response.ResponseMessage == null) return null;
@@ -141,7 +141,7 @@ static HttpMethod AsHttpMethod(Method method)
 #if NETSTANDARD
             Method.Patch => new HttpMethod("PATCH"),
 #else
-            Method.Patch   => HttpMethod.Patch,
+            Method.Patch => HttpMethod.Patch,
 #endif
             Method.Merge  => new HttpMethod("MERGE"),
             Method.Copy   => new HttpMethod("COPY"),
@@ -157,11 +157,11 @@ public static RestResponse ThrowIfError(this RestResponse response) {
 
         return response;
     }
-    
+
     public static RestResponse<T> ThrowIfError<T>(this RestResponse<T> response) {
         var exception = response.GetException();
         if (exception != null) throw exception;
 
         return response;
     }
-}
\ No newline at end of file
+}


Test patch:
diff --git a/test/RestSharp.Tests.Integrated/DownloadFileTests.cs b/test/RestSharp.Tests.Integrated/DownloadFileTests.cs
index fe7a2fde3..aa30a158e 100644
--- a/test/RestSharp.Tests.Integrated/DownloadFileTests.cs
+++ b/test/RestSharp.Tests.Integrated/DownloadFileTests.cs
@@ -7,7 +7,8 @@ namespace RestSharp.Tests.Integrated;
 public sealed class DownloadFileTests : IDisposable {
     public DownloadFileTests() {
         _server = HttpServerFixture.StartServer("Assets/Koala.jpg", FileHandler);
-        _client = new RestClient(_server.Url);
+        var options = new RestClientOptions(_server.Url) { ThrowOnAnyError = true };
+        _client = new RestClient(options);
     }
 
     public void Dispose() => _server.Dispose();
@@ -46,6 +47,13 @@ public async Task AdvancedResponseWriter_without_ResponseWriter_reads_stream() {
         Assert.True(string.Compare("JFIF", tag, StringComparison.Ordinal) == 0);
     }
 
+    [Fact]
+    public async Task Handles_File_Download_Failure() {
+        var request = new RestRequest("Assets/Koala1.jpg");
+        var task = () => _client.DownloadDataAsync(request);
+        await task.Should().ThrowAsync<HttpRequestException>().WithMessage("Request failed with status code NotFound");
+    }
+
     [Fact]
     public async Task Handles_Binary_File_Download() {
         var request  = new RestRequest("Assets/Koala.jpg");
@@ -76,4 +84,4 @@ public async Task Writes_Response_To_Stream() {
 
         Assert.Equal(expected, fromTemp);
     }
-}
\ No newline at end of file
+}