Do HTTP/HTTPS servers ever enforce multipart for file downloads? - Page 1
Forum section: electronics / general

author: peter-h
author_rank: Super Contributor
timestamp: August 27, 2023, 05:15:34 pm
content: I posted this elsewhere but few read that section.So far I have managed to find out that this seems to work:You just look for first occurence of \r\n\r\n (4 byte sequence) and then the data comes straight out, binary.> $ telnet xxxxx.com 80> Trying y.y.y.y...> Connected to xxxx.com.> Escape character is '^]'.> GET /how/img/glide.png HTTP/1.1> Host: xxxx.com>> HTTP/1.1 200 OK> Date: Sat, 26 Aug 2023 14:10:40 GMT> Server: Apache/2.4.41 (Ubuntu)> Last-Modified: Sun, 06 Apr 2014 08:20:34 GMT> ETag: "hghghghghghg"> Accept-Ranges: bytes> Content-Length: 7901> Content-Type: image/png>> <89>PNG^Z (...)That is easy. It would be great if one was able to avoid the multipart delimiters, because they need parsing which is messy on an embedded client; needs a bit of RAM.Are there any standards for this stuff?

author: ejeffrey
author_rank: Super Contributor
timestamp: August 27, 2023, 10:01:51 pm
content: Under normal circumstances a http(s) server will not send a multipart response to a client. Multipart is normally only used for client to server form submission with files via the POST method.

author: HwAoRrDk
author_rank: Super Contributor
timestamp: August 27, 2023, 10:28:45 pm
content: The only common, standard situation I'm aware of where a server will issue a multipart response to a client is when a request has explicitly been made for multiple byte ranges using aRangeheader.Other than that, I suspect an HTTP server would only do so if a specific URL was made to do so for a proprietary purpose. You're highly unlikely to encounter that in the wild though. In fact, I suspect such a server would only give you a multipart response if you gave it anAcceptheader containingmultipart/*(or something more specific).

author: golden_labels
author_rank: Super Contributor
timestamp: August 27, 2023, 10:39:20 pm
content: A multipart response doesn’t require more memory to process than any other response, in practical terms. I’d say this renders the entire question dismissible.A server may respond with multipart data. Realistically this is only going to happen if you already expect receiving a multipart response. HTTP services may be roughly divided into two groups: these meant for web browsers, and API endpoints. So we deal with the following scenarios:A browser making a simple request for normal content:The browser expects a single object in response and it makes no sense for a server to send multiple objects. Browsers couldn’t even handle that.A browser makes a request to a dynamically changing resource:Primarily to webcams sending low framerate images stream over HTTP. The response is realized overmultipart/x-mixed-replace. You must make a request to that kind of a resource to receive a multipart response.A client makes a request for multiple byte ranges:Aside from not very wide adoption of this feature, the client must make an explicit request for multipart data. Which implies it expects the multipart response and can handle it.API endpoints:Rarely encountered, but possible. If you make a request to a particular API endpoint, you already know what kind of response you are going to receive. So a multipart response is possible only, if you already wanted one.Making a plain request without expecting multipart data and recieving a multipart response would be more than weird.

author: peter-h
author_rank: Super Contributor
timestamp: August 29, 2023, 06:22:34 am
content: Thank you all. This has been extremely interesting and useful!

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 07:08:15 am
content: Quote from: peter-h on August 27, 2023, 05:15:34 pmNot really; only internet RFCs, which are the next best thing to an internationally accepted standard. Seehttpwg.org specifications.The one youneedto follow currently isHTTP 1.1 as specified in RFC 9112, because that's what most browsers will use. The only transfer coding you are required to support ischunked.While almost everything still supports HTTP 1.0 as specified inRFC 1945, for any product/application intended for interoperability with generic clients, I do recommend shifting to HTTP 1.1.At least nowadays both browsers and servers send correctly formed MIME multipart data queries (POST file upload) and responses (various, as mentioned in previous messages). I did web stuff back when MS fucked with everybody and had severe bugs in Internet Exploder they refused to fix (no CRLF after latter multipart boundary, keep-alive bugs), and although workarounds were not too onerous, finding out about them was; especially because a majority of users believed that Microsoft could not be wrong, therefore everyone else were just doing it wrong and needed to conform... I mean, even desktops tended to have 16 MiB to 64 MiB of RAM at that time, before the turn of the century. Things could be much worse, really.(That's not to say things couldn't be much better, with a HTTP and HTML/XML-compatible protocol you could parse with a simple state machine, while still being human-readable and -writable. That is one of the topics I do still like to rant about.)

author: peter-h
author_rank: Super Contributor
timestamp: August 29, 2023, 08:09:59 am
content: QuoteThis is for downloading files from web servers. It is a client, HTTPS.Specifically it is for downloading the latest cacert.pem file which comes from here:https://curl.se/docs/caextract.html

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 08:42:59 am
content: Quote from: peter-h on August 29, 2023, 08:09:59 amI'd use a web server under my own control. It would cache the cacert.pem, and serve it using a server configured to return it in a HTTP/1.0 -compatible form, i.e.Code:HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 221470\r\nContent-Type: application/x-pem-file\r\nDate: Tue, 29 Aug 2023 08:30:45 GMT\r\nLast-Modified: Tue, 22 Aug 2023 03:12:04 GMT\r\nCache-Control: max-age=1800\r\nExpires: Tue, 22 Aug 2023 04:07:46 GMT\r\n\r\n... 221470 byte PEM file omitted for brevity...That way, if the URL changes, only the web server configuration needs to be updated. And you won't annoycurl.seeither.Many existing web hosts that cost only a couple of euros per month (I happen to use OVH, with a Lets Encrypt HTTPS sertificate) can be used for this. Just configure the particular URI used in your clients, sayhttps://www.yourserver.domain/ca-certificates.pem, to be provided by a CGI script (or better yet, served as anas-isfile, ie. including its own headers), and have that CGI script emit the contents of the local file as above. Then, set up the local file to be updated (atomically, i.e. using the low-levellink()orrename()method to replace the file when complete) at regular intervals.I'd also let my users reconfigure that URI to something else if they so want, as well as the interval at which to re-read the root certificate chain.

author: HwAoRrDk
author_rank: Super Contributor
timestamp: August 29, 2023, 10:04:51 am
content: Yes, that is a good idea, to cache/proxy the file at a domain and URL thatyoucontrol.Not only is it polite to take the load off the source (especially if there are going to be thousands of devices out there updating at frequent intervals), but you can mitigate any changes in location or availability of the source.Although, on the other hand, I suppose for SSL root certs there is something to be said for "getting them straight from the source" (cURL project not beingthesource, of course, but a trusted source).

author: peter-h
author_rank: Super Contributor
timestamp: August 29, 2023, 10:43:46 am
content: QuoteThere is a trust issue with that, from the customer's POV. If you can corrupt the cert store, all bets are offAnd the mfg (me) would have to maintain that server for ever, a tenner a month in reality, but there isn't a charging mechanism in place.QuoteIt would be perhaps once a month.The URL will be customer-configurable in a config file, accessible over USB.Which takes us back to "security = 0 without physical security" but this futureproofs that aspect of the product.QuoteExactly. Otherwise the whole business of peer authentication using certificates is bogus. You just need to install a fake DNS server and it's all yours

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 11:25:30 am
content: Quote from: peter-h on August 29, 2023, 10:43:46 amIf they don't trust the manufacturer, why would they trust the device? Besides, I already described that the URLs and what they provide must be easily verifiable by the end user anyway.In practice, you are part of the trust chain. Externalizing that node to e.g.curl.seonly makes the trust chain weaker, not stronger, and may piss offcurl.semaintainers.For a wider view of what can happen, look up NTP. There are vendors who simply pick a subset of NTP servers – often Linux distributions' own NTP servers that have nothing to do with that vendor or the OS the vendor uses –, and hardcode the server IP or name into their product, sometimes overloading the NTP servers so much that the distro or NTP server admin changes the name and/or IP address, breaking network time updates for that vendor, who then blames the NTP server for "breaking things". It's not a good thing, and you don't want to cause that kind of ruckus.At minimum, you'd need to askcurl.sefor permission.Quote from: peter-h on August 29, 2023, 10:43:46 amThe upstream source for the Mozilla CA Certificate List ishere at Mozilla Wiki, specifically thePEM of Root Certificates in Mozilla's Root Store with the Websites (TLS/SSL) Trust Bit Enabled (TXT)link. The Wiki page includes a FAQ question on whether/how one can use the CA list. You can use OpenSSL command-line tools on the server to parse the PEM data, to provide the same file or similar as thecurl.sepage does.Quote from: peter-h on August 29, 2023, 10:43:46 amThe "tenner a month" (it's not, it's more like a pound or two per month, and you can pay several years up front) is and should be included in the cost of the device. I'd put any updates et cetera on the same server, possibly behind a customer-specific download key (similar to username-password pair) required for each download, that also provides an unified log of downloads and attempts (use the database the web host provides for this).It is the cost of doing Internet-capable development kits, really.

author: peter-h
author_rank: Super Contributor
timestamp: August 29, 2023, 11:36:20 am
content: QuoteThat is in the future somewhere, and it isn't trivial. I think we had a thread on how to do OTA firmware updates. For example one needs to exclude large volume users (or limit them to older versions) because if one of them gets everything bricked, their lawyers can destroy your company.