Repository: googleapis/google-http-java-client (java)

Issue:
Default character set for text/csv to UTF-8 when absent in the response header.
Some servers don't return charset in the Content-Type HTTP header response. 

[Current implementation](https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/http/HttpResponse.java#L538) defaults to ISO_8859_1 in the absence of charset. 

ISO_8859_1 does not encode unicode characters correctly.

https://www.iana.org/assignments/media-types/text/csv and https://lists.w3.org/Archives/Public/public-csv-wg/2014Oct/0114.html 
indicate that in the absence of charset UTF-8 should be the assumed default. 

#### Environment details

1. General
2. OS type and version: All 
3. Java version: 11 
4. google-http-client version(s): latest 

#### External references such as API reference guides
https://www.iana.org/assignments/media-types/text/csv 
https://lists.w3.org/Archives/Public/public-csv-wg/2014Oct/0114.html 





Pull request description:
fix: default charset to UTF-8 for text/csv if not specified
Some servers don't return the charset. This causes german
characters to be encoded incorrectly, since ISO_8859_1 does not
work very well in such cases defaulting to UTF-8 if its missing.

https://www.iana.org/assignments/media-types/text/csv

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
- [x] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/google-http-java-client/issues/new/choose) before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)

Fixes #1421  ☕️


Patch:
diff --git a/google-http-client/src/main/java/com/google/api/client/http/HttpResponse.java b/google-http-client/src/main/java/com/google/api/client/http/HttpResponse.java
index 2ffa4652..f7bf0b42 100644
--- a/google-http-client/src/main/java/com/google/api/client/http/HttpResponse.java
+++ b/google-http-client/src/main/java/com/google/api/client/http/HttpResponse.java
@@ -534,6 +534,11 @@ public final class HttpResponse {
         // https://tools.ietf.org/html/rfc4627 - JSON must be encoded with UTF-8
         return StandardCharsets.UTF_8;
       }
+      // fallback to well-kown charset for text/csv
+      if ("text".equals(mediaType.getType()) && "csv".equals(mediaType.getSubType())) {
+        // https://www.iana.org/assignments/media-types/text/csv - CSV must be encoded with UTF-8
+        return StandardCharsets.UTF_8;
+      }
     }
     return StandardCharsets.ISO_8859_1;
   }


Test patch:
diff --git a/google-http-client/src/test/java/com/google/api/client/http/HttpResponseTest.java b/google-http-client/src/test/java/com/google/api/client/http/HttpResponseTest.java
index bed019f4..ef759919 100644
--- a/google-http-client/src/test/java/com/google/api/client/http/HttpResponseTest.java
+++ b/google-http-client/src/test/java/com/google/api/client/http/HttpResponseTest.java
@@ -68,6 +68,7 @@ public class HttpResponseTest extends TestCase {
   private static final String VALID_CONTENT_TYPE = "text/plain";
   private static final String VALID_CONTENT_TYPE_WITH_PARAMS =
       "application/vnd.com.google.datastore.entity+json; charset=utf-8; version=v1; q=0.9";
+  private static final String VALID_CONTENT_TYPE_WITHOUT_CHARSET = "text/csv; version=v1; q=0.9";
   private static final String INVALID_CONTENT_TYPE = "!!!invalid!!!";
   private static final String JSON_CONTENT_TYPE = "application/json";
 
@@ -194,6 +195,32 @@ public class HttpResponseTest extends TestCase {
     assertEquals("ISO-8859-1", response.getContentCharset().name());
   }
 
+  public void testParseAsString_validContentTypeWithoutCharSetWithParams() throws Exception {
+    HttpTransport transport =
+        new MockHttpTransport() {
+          @Override
+          public LowLevelHttpRequest buildRequest(String method, String url) throws IOException {
+            return new MockLowLevelHttpRequest() {
+              @Override
+              public LowLevelHttpResponse execute() throws IOException {
+                MockLowLevelHttpResponse result = new MockLowLevelHttpResponse();
+                result.setContent(SAMPLE2);
+                result.setContentType(VALID_CONTENT_TYPE_WITHOUT_CHARSET);
+                return result;
+              }
+            };
+          }
+        };
+    HttpRequest request =
+        transport.createRequestFactory().buildGetRequest(HttpTesting.SIMPLE_GENERIC_URL);
+
+    HttpResponse response = request.execute();
+    assertEquals(SAMPLE2, response.parseAsString());
+    assertEquals(VALID_CONTENT_TYPE_WITHOUT_CHARSET, response.getContentType());
+    assertNotNull(response.getMediaType());
+    assertEquals("UTF-8", response.getContentCharset().name());
+  }
+
   public void testParseAsString_jsonContentType() throws IOException {
     HttpTransport transport =
         new MockHttpTransport() {