DateTime: as usual Randall Monroe is clued up - Page 1
Forum section: electronics / general

author: tggzzz
author_rank: Super Contributor
timestamp: December 14, 2023, 11:42:19 am
content: This will be appreciated by everyone that has had to explain to someone why dates and times arerecursivelymore difficult than most people "think" (and I use that word loosely).Don't believe me? Start by asking them how many seconds there are in a minute, how many hours there are in a day, and how many months there are in a year. I didn't realise the relatavistic differences, and I'm not sure how I could observe themThe caption is "It's not just time zones and leap seconds. SI seconds on Earth are slower because of relativity, so there are time standards for space stuff (TCB, TGC) that use faster SI seconds than UTC/Unix time. T2 - T1 = [God doesn't know and the Devil isn't telling.]"

author: nctnico
author_rank: Super Contributor
timestamp: December 14, 2023, 11:45:55 am
content: Not just for space. Data in fiber optic connections has different speeds depending on whether the signal travels with the earth's rotation or against it. For some systems this matters enough that it needs to be compensated for.In general I loath dealing with date & time in software but somehow I ended up involved with the highest end large scale time&frequency distribution systems.

author: coppice
author_rank: Super Contributor
timestamp: December 14, 2023, 11:58:29 am
content: I used to take recordings of sensor data recorded on aircraft, and run them through simulations of back end systems at a fraction of real time - typically 1/8th, 1/16th or 1/32nd, depending how fast we could make our simulation. So, the concept of "real time" became vague. Was real time the sensor recording domain, or our replay domain? We settled on jargon based on the movie "This Island Earth". "10 our our Earth minutes" became time on the clock in the room, while "10 minutes of real time" became 10 minutes on the aircraft. Get your jargon right, and you are good to go.

author: tggzzz
author_rank: Super Contributor
timestamp: December 14, 2023, 12:05:15 pm
content: Quote from: coppice on December 14, 2023, 11:58:29 amGet your jargon right, and the next subtle "issue" will reveal itselfRecursively

author: tggzzz
author_rank: Super Contributor
timestamp: December 14, 2023, 12:09:34 pm
content: Quote from: nctnico on December 14, 2023, 11:45:55 amAnd for some applications the speed of light in optical fibres is too slow. Hence the High Frequency Trading mob buying up the microwave towers between Chicago and New York, to save a few milliseconds propagation delay. But that mob also encode tradingbusiness rulesin FPGA hardware, to keep latency low.For example...In terms of niche jobs for developers in the financial services industry, it’s hard to get more niche than roles for engineers who work on ultra-ultra low latency systems at systematic hedge funds and high speed market making firms. They don’t come up often, and when they do, they require a special talent: the ability to program field-programmable gate arrays (FPGAs). Both quantitative hedge funds and high speed market making firms hire FPGA engineers. In finance, they’re found everywhere from Citadel Securities, to DE Shaw, Susquehanna International, Maven Securities and Jump Trading. Few come from a financial services background: most are drawn from chip design and hardware engineering firms. Ben Hodzic, executive director specializing in quantitative analytics at recruitment firm Selby Jennings in New York, says they can be hard to come by: there simply aren’t that many FPGA programmers to go around. “Most trading firms will consider FPGA talent from out of industry, given how hard they are to find and how niche of a skill set it is.”https://www.efinancialcareers.com/news/2021/07/fpga-engineer-salaries-hedge-funds-market-makers

author: tom66
author_rank: Super Contributor
timestamp: December 14, 2023, 12:12:13 pm
content: Quote from: tggzzz on December 14, 2023, 12:09:34 pmI believe they're actually using ionospheric transmission now at very low frequencies.A colleague of mine worked on a HFT trading system where the trading decision could be made on the basis of receiving the first 20 bytes out of the 200 byte packet, because that gave some better arbitrage. Apparently they'd get it wrong now and then but it won more often than not that it was worth making the decision based on less data.

author: tggzzz
author_rank: Super Contributor
timestamp: December 14, 2023, 12:17:25 pm
content: Quote from: tom66 on December 14, 2023, 12:12:13 pmWouldn't that imply a very low data rate, and corresponding high latency? My almost negligible understanding is that HFT is based on very large numbers of trades plus cancellations or reverse trades.QuoteGiven correlation with previous trade messages, I can sort of believe how that might work.

author: tom66
author_rank: Super Contributor
timestamp: December 14, 2023, 12:29:43 pm
content: Quote from: tggzzz on December 14, 2023, 12:17:25 pmMy understanding is it provides better overall latency because they can eliminate the repeaters required by a microwave transmission network. Microwave transmission is only line of sight, so it requires periodic repeaters, which each add latency and cost. Ionospheric transmission doesn't require this and also works for cross-Atlantic trades so you can beat conventional fibre HFT between London/Frankfurt stock exchanges and New York/Chicago.The data rate is certainly low. This Bloomberg article suggests it would be limited to individual shares or market elements like Fed rate changes.https://archive.ph/BpMMx#selection-4967.97-4973.544

author: coppice
author_rank: Super Contributor
timestamp: December 14, 2023, 12:37:16 pm
content: Quote from: tom66 on December 14, 2023, 12:29:43 pmLet's see. Balloons at each end to increase the horizon distance. Reflectors on each balloon, to avoid slow transmission up a cable or repeater delays. Chicago to New York is 800 miles. I think that means the balloons need to be about 30km up. Now we just need to decide how to stabilise their positions and we have a plan.

author: coppice
author_rank: Super Contributor
timestamp: December 14, 2023, 12:39:35 pm
content: Quote from: tom66 on December 14, 2023, 12:29:43 pmIf both ends of the link pre-agree on a list of possible actions, they might only need to exchange a couple of bytes of data + some error checking. However, if the data rate is really low even such small messages might represent a latency problem.

author: themadhippy
author_rank: Super Contributor
timestamp: December 14, 2023, 02:04:39 pm
content: Quoteso it wasnt a weather balloon causing problems earlier this year,it was a chinese stock trader .

author: Nominal Animal
author_rank: Super Contributor
timestamp: December 14, 2023, 03:38:12 pm
content: Time is not something we can measure directly, onlyelapsedtime; and in different locations, between any two events, the elapsed time varies. In some extreme cases, even the observed order of cause and effect can be reversed.There are different classes of problems stemming from this. The nastiest one, I believe, is synchronization: consider GPS, or communication with space probes.There are severalclocks, or timekeeping bases, we use here on Earth. The most common is UTC orCoordinated Universal Time, based on TAI orInternational Atomic Time, with leap seconds added or subtracted as needed. (Thus, one minute can be 59 to 61 seconds long, even though almost all minutes are 60 seconds long.)Unix timeas used on most computers is based on UTC.It is notable that TAI is the average of over 450 atomic clocks all over the Earth, so it does not represent the elapsed time on any point, but theaverage elapsed time on the surface of Earthsince a presetepoch, that epoch itself purely notional (not physically verifiable or measurable at all).To convert from UTC to local time, you need two databases: one is the leap second adjustment table (identifying when leap seconds were added or removed), and the other is the local time database including daylight saving times. You need the leap second adjustment table even for counting the exact number of seconds between two UTC date-times.All this means is that our timekeeping is not, and cannot beexact.Realizing this helps, because then you start thinking about what kind of precision isuseful. And then, the timekeeping related problems start falling into different categories (depending on how important they are to the problem at hand), becoming just another numerical/physical (as opposed to algorithmic/theoretic) problem to be solved.In numerical physical simulations with discrete time steps – involving anything from individual atoms, to stars and galaxies –, one key "trick" is to dynamically adjust the time step size to the scale at which events can be represented at useful precision. (It makes it MUCH harder for us humans to observe the time evolution of the system, though; our brains cannot really deal with time "speeding up" or "slowing down".)In many cases from programming to humans waking up from sleep, the exact moment something is scheduled to happen in the future is less important than that moment beingsuitable. (You want to wake up from non-REM sleep, because waking up from REM sleep leaves you groggy. You don't want your computer to do housekeeping stuff like checking for updates while you're working or watching media full-screen, because it leads to jerkiness or added latencies.)All this leads to moving away from "do X at time Y", into "do X after time Y1 but before time Y2": moving away from specificmomentsor points in time, into timeintervals.When you start really working with different aspects oftime, it starts changing the way you think about it. (In a good way, I believe.)

author: tggzzz
author_rank: Super Contributor
timestamp: December 14, 2023, 03:58:07 pm
content: Quote from: Nominal Animal on December 14, 2023, 03:38:12 pmIn a distributed system there can be no common concept of time, and the best you can achieve is partial ordering.Leslie Lamport's seminal papers had something to say about that back in 1978http://research.microsoft.com/users/lamport/pubs/time-clocks.pdfHe won a Turing Award for that!https://amturing.acm.org/award_winners/lamport_1205376.cfmAs for dates, realise that (amongst manymanywierdnesses):in the UK the date 5th September 1752 simply does not existyears have not always started on January 1st, even in Europe. 1751 in the UK (and the eastern bits of what has become the USA) was only 282 days long I first came across this when I was about 10yo. I read a story where some kids were able to prove a document was a forgery because it was dated 5th September 1752.

author: bill_c
author_rank: Regular Contributor
timestamp: December 14, 2023, 04:02:54 pm
content: Tom Scott about date and time.

author: Nominal Animal
author_rank: Super Contributor
timestamp: December 14, 2023, 04:59:35 pm
content: Quote from: tggzzz on December 14, 2023, 03:58:07 pmExactly!Authentication on the interwebs is a perfect example. From the server point of view, there is no ordering to the requests a specific client makes, and they can even be interspersed (as in more than one concurrent page load, with additional requests loading associated files, all at the same time). You cannot update the authentication information on each request, because each response can generate more than one request, and there is still no ordering. You can keep all information on the server, so that the authentication cookie is just a large random number with zero information in it, but that means servers need to share the same active authentication database. Instead, some kind of time-dependent field is used in the cookie.Because there is no common concept of time, regardless of the validity duration of the time-dependent field (say, if it is in minutes, hours, or days, or anything in between), the server side has to be ready to accept the previous and the next value of said field –– either both, or previous for the first half of the validity interval, and next for the latter half of the validity interval.There is no "today"; one must consider "yesterday", "today", and "tomorrow" concurrently.Then, you add to the fact that sometimes, it is possible for the client IP (IPv4 especially, much less so for IPv6) address to change during an authorized session. How do you make sure it is not just an authentication cookie hijacking? If you include the client address or hash of the client address in the authentication cookie, you can detect when that (IP address change, or cookie replay/hijack attack) happens. Time, or order of events is not something you can really rely on, but a backup cookie that has not been used since authentication (limited by cookie path) can be of use. (If an attacker has the entire client browser state, they're in full control of the session anyway.)Ones normal sequential view of time just doesn't work here; one must really understand intervals and how event ordering can vary.And, whenever dealing with secure stuff, how event ordering can be exploited. CVE-1999-0035 is a good example of that: the good ol' ftpd SIGINT bug, which allowed superuser file access to a server just by sending Ctrl+C at the right moment, because ftpd developers didn't realize what might happen if the signal was delivered at just the right moment.

author: tggzzz
author_rank: Super Contributor
timestamp: December 14, 2023, 08:18:50 pm
content: I've occasionally wondered how fleabay manages to be "fair" and "accurate" at the end of auctions, especially with sniping from different continents. I've no real way of telling whether it actually does possess those properties.The HFT mob has timescales a thousand times worse, and they have the inclination and money to be sure

author: coppice
author_rank: Super Contributor
timestamp: December 14, 2023, 08:22:46 pm
content: Quote from: tggzzz on December 14, 2023, 08:18:50 pm"fair" and "accurate", or just "fairly accurate"? I expect fairly accurate is good enough. Most of the latencies are beyond their control, so they really can't do better. It be happy if e-bay just sorted out their broken language selection system.

author: nctnico
author_rank: Super Contributor
timestamp: December 14, 2023, 09:04:56 pm
content: Quote from: tggzzz on December 14, 2023, 08:18:50 pmIs there a rule for auction sites? My guess is that Ebay's software just doesn't care and rounds timestamps to the nearest second. It simply is not a problem.

author: tggzzz
author_rank: Super Contributor
timestamp: December 14, 2023, 09:29:45 pm
content: Quote from: nctnico on December 14, 2023, 09:04:56 pmI haven't noticed a problem, which is a very weak statement.I haven't seen reports of a problem, which is a slightly less weak statement.

author: nctnico
author_rank: Super Contributor
timestamp: December 14, 2023, 09:56:37 pm
content: Quote from: tggzzz on December 14, 2023, 09:29:45 pmOnce again you are trying to create a problem which just isn't there. First come up with a legal requirement that states how auctions are supposed to be held timing wise. If that exists, you have a framework you can build a problem definition upon. But if that isn't there, there simply is no problem because there are no requirements. IOW: a problem can not exist without requirements.If you go to Ebay's website and read about auction timing, you'll see they specify the time an auction runs in units of 1 minute. Basically it means whatever works best for Ebay goes.

author: thm_w
author_rank: Super Contributor
timestamp: December 14, 2023, 10:07:47 pm
content: It doesn't really matter with Ebay, the person who bids the highest before the time ends will win. The point of sniping is another bidder won't have time to go back to their computer and type in a new higher number, so anything 5-10s before the end of the auction should be plenty. Its almost better if its on the higher end as if two people bid $100 the earlier bid wins.Sniping services will let you choose a buffer, and recommend an absolute minimum of 3 seconds for network/processing delays. Good services will also submit from multiple servers at the same time. Which tells me ebays server relies on its own internal timekeeping and doesn't care about what timestamp you have on your packet.

author: tggzzz
author_rank: Super Contributor
timestamp: December 14, 2023, 10:41:27 pm
content: Quote from: nctnico on December 14, 2023, 09:56:37 pmOnce again you are over-interpreting individual words, thus seeing trees and missing the wood under discission.Even trivial experiments using a single browserwithoutusing sniping services demonstrate that the granularity of specifying an auction's end is irrelevant to the points being discussed.

author: tggzzz
author_rank: Super Contributor
timestamp: December 14, 2023, 11:00:08 pm
content: Quote from: thm_w on December 14, 2023, 10:07:47 pmThat's one reason not to bid a round number.QuoteThat raises the question of how multiple competing sniping services interact.I haven't used a sniping service, so am unaware of the available parameters. Using multiple servers is an obvious optimisation, but I'm not clear about the precise benefits.Using multiple browsers and/or windows pointed at the same auction can show "seconds to go" values differing by up to a second (?or slightly more?). It is unclear to me what happens if you somehow manage to submit a bid when one window shows 0s but another shows 1s.It is also unclear to me what correlation there is or isn't between the "seconds to go" values on different computers, and if/how eBay uses that information.

author: thm_w
author_rank: Super Contributor
timestamp: December 14, 2023, 11:40:55 pm
content: Quote from: tggzzz on December 14, 2023, 11:00:08 pmThe benefit is if one server can't connect, power dies, packet gets dropped, packet gets hung up, whatever, then you'll still have another packet that can go through.QuoteIf the PC time is off and you submit the bid late, you would just get an error message.After googling I see a lot of complaints that ebay leaves off the ending time in seconds in some places, so the auction might end at :00 or :59 which confuses them. Also there seems to be a recent issue that if your PC time is too fast, it can cause bidding submit problems (so your browser will think the auction has ended when it hasn't).

author: SiliconWizard
author_rank: Super Contributor
timestamp: December 14, 2023, 11:46:39 pm
content: It's all relative.