"tftp" seems like an incomplete protocol: hacked - Page 1
Forum section: electronics / general

author: DiTBho
author_rank: Super Contributor
timestamp: June 08, 2025, 01:55:37 pm
content: I am developing a bootloader, trying to mimic the behavior of the old Tektronix x11 terminals, which boot from the network as "diskless" computer.It's 90s and early 2000s hype. It made sense.Anyway, for this stuff you need a protocol to "transfer" files from a remote server.tftp is "ok" for this, but ... it has several problemsimplementation problem: tftp-client usually don't return "ERROR" if invoked from a C program via execve or similar. I tried two clients, both print "file not found" or "network problem during transfer", but then they return "0", which means "success", so the ivoking procedure "thinks" all is ok, and go ahead on next stepimplementation problem: I tried two tftp-clients, both implement "file create" (file open with w+) before the transfer, so if you request a file that doesn't exist on the server, you end with an empty file on your clientdesign problem: tftp doesn't have a mechanism to retrieve the file-size from the server, it's simply a not supported feature by the protocol, so when you request a "RRQ" (remote read request) to the server, you get the requested file, which can be 1byte, 0byte, or 1Terabyte, you will only know at the transfer end.My bootloader is a bare metal application, so I don't have any fork() mechanism.So I designed and implemented my tftp client as a library module, to be integrated into the main body.However, it can be compiled as a "GNU/linux" application.Of course, I also designed and implemented the server, this time only as a "GNU/linux" application.I was tempted to add "opcode 6, Size ReQuest (SRQ)", but I tried to be "backwards compatible" with tftpboot, so I implemented it as described in its spec, but added some changes to both the client and the server.Code:opcode operation1 Read request (RRQ) <----------- remote file transfer, from server, to client2 Write request (WRQ)3 Data (DATA)4 Acknowledgment (ACK)5 Error (ERROR)(defined features)Code:Read request.2 bytes string 1 byte string 1 byte+--------+------------+------+------------+------+| Opcode | Filename | '\0' | Mode | '\0' |+--------+------------+------+------------+------+Code:Size Request, implemented as "special "Read request".2 bytes string 1 byte 1 byte string 1 byte+--------+------------+------+------+-----------+------+| Opcode | Filename | '@' | '\0' | Mode | '\0' |+--------+------------+------+------+-----------+------+the trick is very simple.If you want to know the size of a file, just append the '@' character to the end of the file_name and invoke a RRQ request, the server will either understand the type of request and respond with 4 bytes containing the size (32bit, max 4GByte is OK for tftp), or it will respond with the classic error signal "file not found", which in this case means "feature not supported".This simple trick solves the big problem I had with my bootloader: that is, knowing in advance the size of the remote fileto support the progress barto check if there is enough memory to contain the file (physical 32Mbyte, max allocatable for tftp ~8Mbyte )What do you think?Is it too much of a "work around?"The advantage "should be" that if on the other side of the network the tftp server does not support this trick, the client will not know the size of the requested file, will not be able to use the progress bar, but will still be able to "try" to boot, hoping that the file will fit the allocated memory.I mean, rather than not being able to do anything, because the server does not speak my protocol.

author: madires
author_rank: Super Contributor
timestamp: June 08, 2025, 02:19:44 pm
content: Long time ago in a different universe many network elements needed tftp for transferring firmware updates and configurations. A simple, insecure, and cumbersome protocol. Good riddance!

author: Bud
author_rank: Super Contributor
timestamp: June 08, 2025, 02:36:51 pm
content: The word "Trivial" in tftp protocol name is there for a reason.

author: ejeffrey
author_rank: Super Contributor
timestamp: June 08, 2025, 06:06:03 pm
content: tftp only really makes sense for extremely limited environments. The goal is a tiny amount of code and generally no error recovery is possible. The "problem" that a client executable doesn't return the right error code to the shell is hardly a problem as normally a stand alone executable wouldn't be used. Normally it's not even used with a filesystem. Rather it's a function available from a boot ROM.These days even the most constrained boot environment probably has the resources for something more but it sticks around because these things never die and it generally does the job.It's got no security of its own but if you care it's used with image signing -- which applies wherever the image is loaded from.I terms of your extension: does the transfer take long enough that a progress bar matters? What do you do if the file is too big for memory? Is that better than simply starting the transfer and failing when full? You will need to check the size during the transfer anyway, is an early failure better?

author: DiTBho
author_rank: Super Contributor
timestamp: June 08, 2025, 06:36:40 pm
content: 9Mbyte of kernel and 16Mbyte of initrd over a 10Mbps link.Say 900kbyte/sThe progress bar avoids people switch off the device thinking it's blocked.

author: DiTBho
author_rank: Super Contributor
timestamp: June 08, 2025, 06:38:21 pm
content: If the file is too big, the bootloader allows you to choose a different profile, which has a smaller initrd.

author: DiTBho
author_rank: Super Contributor
timestamp: June 09, 2025, 02:41:52 pm
content: Quote from: Bud on June 08, 2025, 02:36:51 pm"trivial" means "it provides simple communication" with the client for sending and receiving the files in the form of packets.And it was not my choice to use "tftp" as "booting protocol" for Tektronix Xterms (1990s) and modern (2025) u-boot-based stuff.It's there, and you need to support it.I need to extend it.

author: DiTBho
author_rank: Super Contributor
timestamp: June 09, 2025, 02:47:13 pm
content: Alternative, "ftp"?They { "ftp", "tftp"} are both used for transferring files from one system to another.The difference between both is that "ftp" provides more security (login?, permission?, ... ) and allows complex functions (list, mput, mget, ...), commands, and operations on data whereas "tftp" does not provides security features and complex commands."ftp" consumes more resources, anyway.

author: DiTBho
author_rank: Super Contributor
timestamp: June 09, 2025, 02:56:26 pm
content: If you have a GNU/Linux-capable SBC, then there is also "sftp".You need to "transfer files", so why not use the "Secure File Transfer Protocol"?It is based on OpenSSH, and you can consider it as an advanced version of the "file transfer protocol", but it also provides security (via encryption) when transferring files between organizations/computers.Just reserve and keep open port 22, and and use the SSH-client-server model ...The problem is: everyone is good at talking in Forums, but ?!?who?!? is willing to implement from scratch this thing in a firmware like u-boot?Nobody? Here we go ...

author: ejeffrey
author_rank: Super Contributor
timestamp: June 09, 2025, 08:31:19 pm
content: Quote from: DiTBho on June 08, 2025, 06:38:21 pmMakes sense. I have only used tftp in situations where the transfer took only a few seconds. In that case, trying to download the default file and failing is good enough. If the download is slower then finding out before spending so much time is probably worth it.

author: mariush
author_rank: Super Contributor
timestamp: June 09, 2025, 09:34:03 pm
content: You could just support HTTP 1.1 and be done with it. But I guess this would require support for TCP/IPA client that supports only GET and HEAD requests would be fairly easy to make.You could use Range in the request to get partial content, you could check if http server supports partial transfers by checking for Accept-Ranges in a HEAD request, or in the reply to the first chunkSeehttps://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Rangeandhttps://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-RangesAnother benefit of plain http is that you can add some amount of stuff in the reply that's before the actual data, like for example you could have a custom extension to the http server to give you a CRC32 checksum for the chunk of data you getCode:HEAD /boot.bin HTTP/1.1Host: 192.168.0.101Accept: */*Code:HTTP/1.1 200 OKContent-Length: 1048576Content-Type: application/octet-stream…Accept-Ranges: bytesIf you customize to get checksums your server could addCode:Checksum-Enabled: yesChecksum-Type: CRC32, CRC16, SHA256then your client could request just the first 4096 bytesCode:GET /boot.bin HTTP/1.1Host: 192.168.0.101Accept: */*Range: bytes=0-4095You could add support for gzip decoding and use Accept-Encoding: gzip to tell the web server you support compressed content with gzip, then you also have a CRC32 checksum inside the gzip stream, so no need to customize the responses with custom checksum tags.If you make your requests 32-64-128-256 KB in size, you would be able to easily decompress the chunks with limited amount of memory on the controller.

author: golden_labels
author_rank: Super Contributor
timestamp: June 10, 2025, 06:15:22 am
content: DiTBho: I believe it would help to think of TFTP the same way as of data sent over any internal bus or over wire protocols like RS-232 and RS-485. This is the same idea, just implemented over local IP. It is not a user-facing protocol.mariush: HTTP is supported by DOCSIS devices, as well as by many other network equipment. But this is not suitable for what TFTP is being used for. As you noticed, it requires TCP. But HTTP is also much, much more complex than TFTP. We might make a non-compliant implementation, but then we’re effectively designing a new protocol. So why bother with trying with half-broken-HTTP, instead of using a well tested, widely deployed and supported protocol, designed to do exactly what it is needed? Whic his… TFTP!

author: gf
author_rank: Super Contributor
timestamp: June 10, 2025, 06:53:38 am
content: The obvious aim of TFTP was to be minimalist, running over UDP without requiring a full TCP/IP stack. The packet size is small to avoid the need to implement IP fragmentation and reassembly. Sometimes, if the server's MAC address is known from the preceding BOOTP/DHCP stage, the TFTP client in the bootloader also renounces to use ARP for MAC address resolution.

author: DiTBho
author_rank: Super Contributor
timestamp: June 10, 2025, 09:32:23 am
content: No one commented how I extended tftp, adding get file_size.Do you think it's ok as I did it, or is it better to add a specific "opcode"?By the way, there are 16 bits for 65536 opcodes,but only 5 are defined.Code:opcode operation1 Read request (RRQ) <----------- remote file transfer, from server, to client2 Write request (WRQ)3 Data (DATA)4 Acknowledgment (ACK)5 Error (ERROR)(defined features)5 of 65536, why not 8 bit for the opcode instead?Hmm, I don't understand

author: golden_labels
author_rank: Super Contributor
timestamp: June 10, 2025, 12:24:26 pm
content: DiTBho: my reply is also commenting on your idea, but I suppose I deleted too much out of it to make it clear.For what TFTP is meant to be, I don’t see a reason to extend it. Of course for your own device you can do anything you wish, but I don’t see any benefit in the described extension in the wild.The nature of TFTP suggests that in most cases the sender should be aware ahead of time of file sizes and limits.

author: gf
author_rank: Super Contributor
timestamp: June 10, 2025, 12:58:14 pm
content: Quote from: DiTBho on June 10, 2025, 09:32:23 amDo you really need it?Besides DHCP option 67 (boot file name), there is also option 13 (boot file size) defined in RFC 2132.

author: DiTBho
author_rank: Super Contributor
timestamp: June 10, 2025, 01:18:23 pm
content: Quote from: gf on June 10, 2025, 12:58:14 pmYes! And I explained why!QuoteInstead of extending tftp, you are suggesting to use bootp/dhcp.Code:Boot File Size OptionThis option specifies the length in 512-octet blocks of the defaultboot image for the client. The file length is specified as anunsigned 16-bit integer.The code for this option is 13, and its length is 2.Code Len File Size+-----+-----+-----+-----+| 13 | 2 | l1 | l2 |+-----+-----+-----+-----+

author: DiTBho
author_rank: Super Contributor
timestamp: June 10, 2025, 01:26:48 pm
content: ummmm, no machine here uses a dynamic IP, they are all static.but it's not a problem.Let me think about this

author: gf
author_rank: Super Contributor
timestamp: June 10, 2025, 02:57:59 pm
content: A commonly used PXE boot procedure is: 1) get network parameters and boot file name via BOOTP/DHCP, 2) load the boot file via tftp.But I don't know if that fits your environment...The disadvantage of a proprietary extension is that you still can't boot from an RFC-conforming boot server that is unaware of your protocol extension.

author: DiTBho
author_rank: Super Contributor
timestamp: June 10, 2025, 03:05:34 pm
content: Quote from: gf on June 10, 2025, 02:57:59 pmI think I wrote that the extension I proposeddoes NOT break anything, so you can safely boot from a classic tftp server that does not support my extension, you simply will not have the progress bar and you will have to wait until the end of the transfer or a good percentage of it to know "if it fits in the ram" or if it was aborted.

author: gf
author_rank: Super Contributor
timestamp: June 10, 2025, 03:49:31 pm
content: Sorry, didn't read the thread from the beginning. Responding to a special file name is certainly not a protocol vialation.

author: Nominal Animal
author_rank: Super Contributor
timestamp: June 10, 2025, 04:20:57 pm
content: Quote from: DiTBho on June 10, 2025, 09:32:23 amI would have used '?' instead of '@' myself (but I understand the risk with mixup regarding globbing with*and?), but otherwise your solution is superior, for the simple reason that it is compatible with unmodified servers: those will simply respond with file not found error. It is a common "error", and therefore well tested in TFTP implementations, unlikely to produce any weirdness.The opcodes would be nicer option in that it would not limit file names, but we just don't know how various TFTP implementations handle unexpected opcodes.

author: ejeffrey
author_rank: Super Contributor
timestamp: June 10, 2025, 04:39:01 pm
content: RFC 2349 includes an optional transfer size fieldhttps://www.rfc-editor.org/rfc/rfc2349which requres the option extensionhttps://www.rfc-editor.org/rfc/rfc2347It looks like you send a read request with the transfer size field set to 0 and the response option-ACK has the actual size included. tftpd write also has a size field which allows file servers that permit uploads to reject files that are too large.hpa's tftpd server man page documents that it implements these extensions:https://linux.die.net/man/8/tftpd

author: DiTBho
author_rank: Super Contributor
timestamp: June 10, 2025, 07:37:48 pm
content: Quote from: Nominal Animal on June 10, 2025, 04:20:57 pmThis is my concern, and looking at the official extensions { RFC2347, RFC 2349 } reported by @ejeffrey, it seems to be problematic that way.It seems that stuff was discussed in 1998, strange that none of the Tektronix XP4xx and NCD4xx and 9xx produced in 1999 and early 2000s have these things implemented. Also weird that a couple of versions of uboot 2009 for PPC4xx do not have the extensions implemented.Dunno, but I can add this stuff.edit: snip

author: ejeffrey
author_rank: Super Contributor
timestamp: June 12, 2025, 06:02:34 am
content: Quote from: DiTBho on June 10, 2025, 07:37:48 pmIt certainly could cause problems, but I would generally not worry about using standard extensions. Especially given it's published over 25 years ago, appears to be implemented in some common clients and servers. It's unlikely you are going to find implementations in the wild that are actually incompatible with the RFC.The risk with adding an opcode is really that someone else will have extended the service in a different way and used the opcode for something incompatible and cause unexpected behavior. RFC2347 appends the options to the end of the read or write request packet. The authors state that they expect most implementations will just ignore that data if they don't know to look for it. In TFTP all the fields are null terminated strings, so this seems reasonable. The options are indicated with names rather than numbers so it's also less likely to be misinterpreted.

author: Nominal Animal
author_rank: Super Contributor
timestamp: June 14, 2025, 05:47:41 am
content: Quote from: DiTBho on June 10, 2025, 03:05:34 pmUnless, of course,They can even get the progress bar ifthey create a parallel file tobootfilenamedbootfile@that contains the size of the actual boot file, in binary.Hmm.. Would that be enough of a reason to change your protocol to pass the size in bytes as a decimal string instead? Allowing zero or more leading ASCII whitespace (\t,\v,\f, space), one or more ASCII digits (0-9), and requiring at least one trailing whitespace including newlines (\t,\n,\v,\f,\r, space)?That way it'd be trivial to create that file, using "stat -L -c '%s' 'bootfile' > 'bootfile@'". (statis part of coreutils, so it will be installed in all Linux systems. It's a common Unix utility. The parsing overhead is neglible compared to the transfers themselves.)

author: DiTBho
author_rank: Super Contributor
timestamp: June 14, 2025, 07:31:31 pm
content: Quote from: Nominal Animal on June 14, 2025, 05:47:41 amI don't think I understand the problemA file ending with "'@" is not valid for the filesystem.Anyway, the trick is not to use a file that contains the filesize.If the server supports my extension, it redirects the handler to a procedure that opens the file, gets the filesize, and sends it to the client in binary format.If the server does NOT support the extension, it returns "file not found".I've checked a couple of servers that do not support the extension. There are no problems.I don't have much time right now, busy with other issues, but as soon as I get back to the tftp, I'll extend the client and server with { rfc2347, rfc2349 }

author: Nominal Animal
author_rank: Super Contributor
timestamp: June 15, 2025, 12:16:17 am
content: Quote from: DiTBho on June 14, 2025, 07:31:31 pmNo problem –– I worded it wrong, now edited to fix! ––, the opposite: you can get even the progress meter back with simplest of TFTP servers.If the size response is the same as reading a file containing the actual file size in bytes, and your own server synthesizes that file on the fly, users can use your client with any TFTP server, without losing functionality. With other TFTP servers they only need to create the size-file by hand. If they don't do that, the progress meter is not shown.That is, when the TFTP client wants to load "bootfile", it first requests file "bootfileSIZE".Your own TFTP server synthesizes its contents, the size of file "bootfile" in bytes as an ASCII decimal string followed by a newline. If "bootfile" does not exist, this file does not exist either. In practice, your TFTP server should always check the file existence first, and then, in the cannot-read case, check the filename suffix and if correct, only then trigger your custom handler.Your client parses the response, and uses that length for the progress bar when loading "bootfile", the proper boot data image file.With any other TFTP server, the user can, but is not required to, to create the "bootfileSIZE" file themselves, using e.g. the commandstat -L -c '%s' 'bootfile' > 'bootfileSIZE'If they do, it will let the client display the progress. If they don't, the progress is not shown.Above,SIZEis a valid but rare or reserved file name suffix. (In Linux, file names are simply non-empty byte sequences not containing 0/0x00 or 47/0x2F; every other value is technically allowed, just not supported by some non-Linux/POSIX/Unix filesystems.)I believe this yields much wider compatibility than any of the other approaches for the server side, including the later RFCs.If you wanted security or proper data integrity checks and connection resiliency, a custom transport on top of UDP using ChaCha20-Poly1305 or AES128-GCM is not difficult to implement. It is really thekey exchange, the handshake part, that is tricky; especially if you want to avoid RSA and Diffie–Hellman (because that is what makes TLS slow on old hardware). It is why I've looked into offloading the annoying parts toMicrochip ATECC608Bvia I²C, specifically using a small board with the IC, a supply bypass cap, and a 4-pin JST PH2.0 connector. To change keys, one just changes the tiny board (BOM < 1€ in singles); works even for IoT MCUs.

author: DiTBho
author_rank: Super Contributor
timestamp: June 15, 2025, 07:22:21 am
content: ah, got it, brilliant

author: westfw
author_rank: Super Contributor
timestamp: June 16, 2025, 05:19:50 am
content: QuoteI like it. Since I was thinking along essentially identical lines.You could request "bootfileMETADATA" and put whatever you want in there.

author: Nominal Animal
author_rank: Super Contributor
timestamp: June 16, 2025, 06:15:19 am
content: Exactly. I only suggested changing the content (synthesized or not) to human-readable decimal string form, because 1) that has no endianness issues, 2) the conversion overhead is absolutely neglible compared to the actual transfers over even the fastest 10G transports even if you used BASIC on an 8-bit computer to implement the TFTP client, and 3) it is veryKISSand understandable to users: a single line explanation suffices to anyone who can set up a TFTP server!And yeah, it definitely could contain more, for example two lines of formsize-in-bytessha256sum-in-base64which would let capable TFTP clients use the size in bytes for progress meter, but also calculate the sha256sum of the transferred firmware file for integrity verification.If you don't want to add encryption but would like to add passphrase-based authentication, you could add third line withsha256hash-of-above-sha256sum-and-the-passphraseso that the sha256 hash of the concatenated sha256sum of the data file and the passphrase forms an unreversible verification that whoever sent the data, knew the passphrase too, intended exactly that data to be received, and no man-in-the-middle fiddled with it.(Obviously, I suggest a line-based form with ASCII contents.)I'm not exactly sure what the "SIZE" or "METADATA" filename part should be. DiTBho has some filename restrictions, and a single character is compact, but I'm leaning towards something like.size, because even if it contained more than just the size, it would be perfectly intuitive way to associate it with the firmware. After all, if you sawmygadget-fw-20250617-v203.binmygadget-fw-20250617-v203.bin.sizeI'm pretty sure you would have a correct intuition about their relationship.A TFTP server that can synthesize (without creating the metadata file) the metadata on-the-fly should probably still use the file if it exists.I don't think I'd even add a modification timestamp or size checks (to "auto-detect" stale metadata files), because, well, KISS.(And note that a synthesizing server can choose to synthesize the size only, the size and the checksum, or the size, checksum, and verification hash, for example based on server configuration. In other words, the above order of these three lines isn't random, it was intentional on my part – although I didn't think of the checksum and verification hash before westfw hinted at such above.)

author: mariush
author_rank: Super Contributor
timestamp: June 16, 2025, 07:38:30 am
content: using .size would make most sense to me, provided ONLY the file size is inside the file ... the parser would be really simple... parse 0..9 and stop at the first character that's not 0..9use maybe .info or some other extension if you want to have more stuff ... could be something simple askey [ = | : ]value [ separator ] (repeat with next key+value pair ) where separator is limited to space, tab, ";" or "," or newline characters \r \n ... ex .. optionally maybe accept " " or ' ' for strings to support those separator characters in strings."size=100, crc32=afafafaf, sha256=af2134..23"Keeps it human readable and writable and editable and is less complicated to parse than JSON or more human readable than bencoding for example.

author: Nominal Animal
author_rank: Super Contributor
timestamp: June 16, 2025, 08:01:04 am
content: I think.infowould work well, but your protocol suggestion is definitely not KISSy. It contains unnecessary fluff.My suggestion for the contents was basicallyLWS*size[LWS+checksum[LWS+authhash] ] LWS+whereLWS = (HT | LF | VT | FF | CR | space). Ordered, withsizerequired,checksumoptional, andauthhashoptional but only allowed ifchecksumis also listed. You know, one to three values, parsing trivial, minimal space overhead.The required trailing whitespace is there to ensure the recipient is absolutely sure the data is complete and was not truncated.If you want a keyed format, simplest is( LWS*keyHWS+valueHWS* NL )+ LWS*whereHWS = (HT | space)andNL = (LF | CR). For example,sizesizesha256checksumauthauthhashBut, then you have to handle the cases wheresizeis not specified, orauthhashis butchecksumisn't, and so on. Not KISSy.

author: westfw
author_rank: Super Contributor
timestamp: June 16, 2025, 06:26:39 pm
content: QuoteNote that ".info" is a file format already in use for providing documentation/manuals for many GNU projects. A sort of 1989-era semi-hypertext formathttps://www.gnu.org/software/emacs/manual/html_node/info/index.html

author: DiTBho
author_rank: Super Contributor
timestamp: June 16, 2025, 10:04:10 pm
content: I have a simple rule: one request, one service.

author: paulca
author_rank: Super Contributor
timestamp: July 09, 2025, 03:38:39 pm
content: Errors? There are no errors. There is no failure. There is only reset, retry. You are either successful or retrying. I have had modems trying to connect for years, they don't seem to mind. Never seem to get bored and error out. Like who they gonna call?

author: bson
author_rank: Unknown
timestamp: July 18, 2025, 08:07:12 pm
content: Quote from: DiTBho on June 08, 2025, 06:38:21 pmBoth the kernel and initrd at least in the past could also be compressed; not sure if that's still the case.tftp is still used, especially by kernel developers. You can have simply add a make target to deposit a bootable image or images in a tftp folder somewhere, then boot it on one or multiple test systems. When I did this at Sun (Solaris kernel) I had a sun4u on my desk, a sun4u to test on, and sun4m and sun4c systems to verify compatibility on. We also supported old versions of the system until EOLed, so sometimes when a critical bug was fixed I'd have to backport the fix to multiple older versions, build and test those. Sometimes the fixes ended up differing substantially. tftp was a godsend for this.The Sun tftp openboot loader produced the classic Sun ASCII spinner.