Low latency, high resolution media player - Page 1
Forum section: electronics / general

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: October 30, 2013, 04:53:58 pm
content: A customer of mine is being quite successful at selling interactive video artworks like this :http://cinimodstudio.com/project/flutter/(which I designed, using playback from NAND flash via a small FPGA)and thishttp://cinimodstudio.com/project/baby-flutter/(based on a beagleboard)and also large-scale stuff like thishttp://cinimodstudio.com/project/deep-blue-interactive-aquarium/(uses very high-end PCs for realtime rendering)We're looking at scaling up the smaller units up to HD type resolutions, and are drawing a blank on suitable hardware platforms. Most video playback slutions are unsuitable as we need minimal latency displaying randomly selected frames at 60fps, so anything with pipeline delays or keyframe+difference compression isn't responsive enough, and high image quality is important, so can't use much compression.In addition to fast interaction, another requirement is fanless operation and reasonable power efficency, so we can;t just throw a turbo-nutter PC at it.My thinking is that instead of looking at mid-high end processors, a better approach is probably to just store uncompressed images and fire them out to HDMI as required with an FPGA.Either a few pages of frame buffer plus a fast SSD, or enough RAM to cache a substantial amount of content (a few Gbytes) and a slower mass storage solution.Or maybe use an SSD with minmal buffer and rely on the SSD buffer/cache for frame refresh.It's not a problem if it takes a while (a minute or two) at startup to load up the RAM.The actual processing will be very low-effort - taking inputs from sensors and deciding which pre-rendered frames to display when.the content would be written once and rarely changed, so the playback system would not need to worry about any writing, and teh media can be formatted to make it easy - e.g. absolute block addressing.The volumes could get into the low hundreds, so a custom solution may be viable - I've had a quick look at various boards but can't immediately find anything with a lot of RAM (or DIMM socket).The hardware I'd envsage would look something like:FPGA (guessing Spartan-6 sort of level), DIMM socket(s) for expandability, SATA or high speed SD card. Low-speed interface (SPI/UART) to a seperate host processor (Raspberry Pi or something comparable) and HDMI/DVI out.Passive cooling - the board could be arranged to suit cooling via a large heatsink and/or heatpipe, but no fan - it must be silent.Looking at RAM bandwidth, we're nowhere near bleediong-edge. at about 250MBytes/sec (1024x1280x24bppx60fps).A build cost in 100x of up to, say £300-ish would be OK.Although it could in principle drive LVDS panels direct, a concern is supporting different connectors etc. on different panels. We're currently thinking that HDMI may be a more useful output as it gives a lot of flexibility, and allows for driving projctors etc.I'd be interested in opinions on :Any off-the-shelf solution that might come close (or be adaptable)As a relative FPGA Newbie, how hard is it likely to be to get an FPGA+memory+DDR+HDMI solution working?I know most FPGA vendors have some solutions to help with memory, HDMI etc., but how easy are these to use in practice? I have managed to get HDMI in/out working on anAtlys board with ready-rolled modules, but only at medium resolution.What might it cost to contract out the hardware and FPGA design to do something like this. I know enough to be able to produce a very well-defined detailed spec, and we can leave all the stuff like filesystem management to the host processor that we would be in charge of implementing - the custom part would simply be a display engine that displays specified blocks on command.

author: nctnico
author_rank: Super Contributor
timestamp: October 31, 2013, 02:24:47 am
content: You don't have to use many keyframes. A lot depends on the encoding. On my PC I can skip forward/backward instantly through most x.264 videos. I'd look into a solution around an iMX5x from Freescale which has hardware decoding to see how much latency it has. I tinkered a bit with an iMX51 board which has an HDMI output and it could play video nicely.edit: IMHO the iMX53 series from Freescale looks powerful enough for what you want to do (I've used the iMX51 in a design). The iMX series does most of the heavy lifting and display handling in dedicated hardware so the CPU and memory bus are free for other processing tasks. The DM3730 on the Beagle board does almost everything in software (its basically an ARM core and a DSP core) which puts constraints on memory bandwidth. I'd investigate going the SoC route before committing to a dedicated FPGA device. Many years ago I designed a data acquisition board which uses standard PC DDR memory (and supports different memory configurations) together with a Spartan 3 FPGA. Getting the 64 bit wide DDR200 memory up&running did take some effort. OTOH 1.5GB/s sustained throuhgput is pretty neat :-)Besides that it may be interesting to look into motion JPEG. This is basically a string of independant JPEG images.

author: Legit-Design
author_rank: Frequent Contributor
timestamp: October 31, 2013, 01:33:01 pm
content: About going fanless: "turbo-nutter PC" without fanshttp://www.quietpc.com/sys-a470http://www.quietpc.com/sys-a480No idea if it's going to cost too much for driving 100 displays, but definitely no need to have 100 actual pc:s driving the displays. Also no need for fully custom hardware and design costs.Reasonable power efficiency depends on compared to what.PC already has everything required right out of the box:RAM bandwidthSata ports for fast ssdDual/triple/quad? HDMI/DVI with additional graphics cards.PC should be able to drive multiple displays, or how critical is the latency for different images for separate displays?Should be a lot easier to have it displaying 60fps pictures with low latency, if that's the final requirement. More or less plug and play. I don't think it actually needs hardly any graphics grunt if pictures is all that's required.Could even boot from memory stick and load images to huge ram, up to 32GB with normal consumer grade MB.I have no experience about those particular systems, but I see no reason why they wouldn't be reliable and all that.I know hobbyists have ran their custom built fanless silent pc:s for years already. Using heatpipes and custom machined copper blocks into big heatsinks.How critical is the fanless requirement? Modern good quality fans can be near silent in most situations. Scythe and Noctua make extremely silent fans. With big heatsinks fans don't need high rpm to sustain adequate cooling. I guess there are other reasons for going fanless, like maintenance or not having to clean the dust filters.Few years ago I had cabin heater from old car as water cooling for processor only. Without any fan for the radiator, rest of the pc still had fans. Cabin heater radiator was around 2x120mm fan sized. It ran fanless at stock speeds and voltages, overvolting for more megahurtz made it unreliable, so it needed more cooling.

author: amyk
author_rank: Super Contributor
timestamp: October 31, 2013, 02:01:58 pm
content: Couldprocedural generationtechniques be used to cut down (static) storage requirements?

author: grumpydoc
author_rank: Super Contributor
timestamp: October 31, 2013, 02:55:06 pm
content: How many frames do you want to store - uncompressed full HD video sucks up quite a bit of storage, at 60fps you're looking at just under 300MB per second of video so even 32G of RAM is only going to hold about 90 seconds.If you compress the video then you either won't need so much RAM or will be able to up the duration (or a bit of both) - I suspect if you want to do it with an FPGA your options are jpeg for each frame, motion jpeg or I-Frame only H.264That said an alternative might be a SoC based system attached to each display which is fast enough and then you can attack the problem with software tools (I presume you have played with ffmpeg) which might make it easier - as you say achieving full HD at 60fps is difficult but if you can drop to 30fps something like a cubieboard2 might be OK (if you have the budget to put one behind each screen).It has 4GB NAND flash - using I-Frame only H.264 (say you might get away with 10MB/sec depending on source material) you could get up to 5 minutes of video with no additional storage. Seeking on an I-Frame only stream is very fast so that would probably be OK for interactivity.If you are happy to design a system then the Allwinner A23 or A31 both claim to be capable of 1080p@60fps but I'm not aware of a pre-built SBC which uses either.Edit: Actually the frames rates quoted above are forencode, now that I check again - decode for the A20 (cubieboard2) is rated at 3840x1080@30fps so 1920x1080@60fps should be possible, so a cubieboard2 attached to each display should be able to do what you want and at less than £60 a pop well within your budget.

author: Stonent
author_rank: Super Contributor
timestamp: October 31, 2013, 06:39:39 pm
content: Will this be true HDMI encryption with audio or just straight raw DVI at 1080p60?

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: October 31, 2013, 11:23:21 pm
content: Quote from: Stonent on October 31, 2013, 06:39:39 pmJust raw video - pre-rendered framed from 3D models. No audio

author: ConKbot
author_rank: Super Contributor
timestamp: November 01, 2013, 04:06:27 pm
content: If going with a PC based solution, look into daisy chaining displayport monitors with displayport 1.2/multistreaming a lot of video cards support upto 6 monitors, and not all super high end gaming cards either.You'd have to check into other restrictions first, but if you can run 6 monitors per card with a triple or quad card system, 18-24 displays per system may be feasible, $800-$1k for cards per system and ~$800 for the rest of the PC. (just SWAG on those numbers though)

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: November 01, 2013, 04:38:04 pm
content: I really don't think a PC based solution is the way to go - you're burning a lot of power for something taht doesn't actually need to do much processing. I suppose it might be worth looking at extreme underclocking, but I suspect anything that has high enough memory and display bandwidth will tend to come with a lot of additional high-speed baggage.the final product wants to look like a framed picture, so needs to be a thin assembly mounted on the back of an LCD panel. We can use a large aluminium backplate for passive cooling, but if it's just an FPGA and some RAM I can't seeing it needing much.

author: daveshah
author_rank: Unknown
timestamp: November 01, 2013, 05:02:58 pm
content: The dual/quad core versions (and possibly the single core version) of the Freescale I.MX6 processor can address up to 4Gbyte of DDR3, which might be useful. They also have built-in HDMI and two LVDS outputs.

author: Marco
author_rank: Super Contributor
timestamp: November 01, 2013, 07:11:58 pm
content: I wouldn't discount H.264, purely for image coding it's still close to the state of the art.There are ARM boards which can decode and outpu 1080p60, but often very high bandwidth videos (such as intra-only) can make decoders stumble so you'd have to test whether they can handle it. With hardware which can handle it you just need to cobble together a transcoder to feed the hardware decoder a single continuous video, most of the code for that is already available open source.Still seems a hell of a lot less work than a FPGA solution and with a hell of a lot better compression than you could manage in it.

author: orion242
author_rank: Unknown
timestamp: November 01, 2013, 07:20:16 pm
content: These guys make industrial PCs, some are nothing more than a panel mountable touchscreen. No fans, low power, and as much horsepower as you want. Prices from $500 to the sky depending on what you want. Used them a few times in nasty environments and was pretty impressed.http://www.advantech.com/default.aspx

author: grumpydoc
author_rank: Super Contributor
timestamp: November 01, 2013, 07:24:15 pm
content: QuoteLightly compressed video is usually easier to handle from the decoder point of view, because there's less work to decode it. Where I have seen problems is that the software has been sloppily written with fixed sized buffers which then overflow with the larger input data (if you think about it the output data is the same size and same bitrate however highly or lightly compressed the input). Slow storage systems which can't deliver the data fast enough to the decoder can also be an issue.I still think that something like a cubieboard2 per display is the way to go.I-frame only H.264 compresses better than straight JPEG or motion JPEG as far as I know and pretty much by definition it will take less than 1/60th of a second to decode a frame and start showing it on the screen.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: November 01, 2013, 09:20:44 pm
content: Quote from: grumpydoc on November 01, 2013, 07:24:15 pmBy definition you can decode at 60fps, but that does not guarantee low latency, as it could be pipelined, or interleaved over multiple parallel hardware accelerators, so the delay between specifying a particular frame and it appearing could be longer - I don't know enough about how these things work to know if this is an issue in practice.

author: grumpydoc
author_rank: Super Contributor
timestamp: November 01, 2013, 09:54:43 pm
content: QuoteI know what you mean but don't think it would be an issue - most of the latency comes from large GOP sizes and the need to process several frames to get at the one you're interested in - and you can get rid of all that with an I-frame only stream.You haven't said how much video you want to store but at 60fps full HD a GB is going to last you three seconds, four if you want 1024x1280 rather than 1080x1920.And cubieboards aren't exactly expensive for you to try it out.Of course I'm more a software guy (at least that's where I have formal training) so would tend to prefer a solution which doesn't involve me designing building hardware from scratch - I suspect your perspective differs somewhat

author: denizcan
author_rank: Regular Contributor
timestamp: November 06, 2013, 11:56:37 am
content: I don't think he needs any compression. The computers are not good for realtime stuff.. At least if you are running windows.. Compression would add some delay, and make thinks complicated. A raw 1920 x 1080 x 32 bit frame is just 8MB. It is really easy and cheap to put enough RAM for 1000s of frames.. This will also make the images super sharp.. No FPGA is needed, even a raspberry pi might be platform of proof of concept.. Loading RAW images from SD card to the RAM, and playing them wouldn't need much effort.. The only problem is the address range a 32 bit MCU supports, however, even extending it does not require an FPGA.. A few logic glue, or maybe CPLD is better..

author: nctnico
author_rank: Super Contributor
timestamp: November 06, 2013, 12:41:17 pm
content: You are forgetting that you need a sustained throughput of 360MB/s and at least some form of double buffering in the frame buffer to avoid flicker. I doubt the Raspberry pi can do either. A 32 bit DDR200 memory system has a peak throughput of 800MB/s. Because the frame buffer is usually in the main memory you already loose half of the memory bandwidth. If you need to move 360MB/s from the main memory to another place in the main memory and display a frame at the same time you'd need over 1080MB/s.

author: grumpydoc
author_rank: Super Contributor
timestamp: November 06, 2013, 02:07:02 pm
content: QuoteI was going to make the same point but you beat me to it - it's the sustained rate that is the problem. You can't pull uncompressed frames off of a SD card fast enough. Even cards which exceed class 10 would only be doing a few frames per second which is some way off Mike's spec.The allwinner A20 has a DDR3 interface which should be fast enough - I think the Cubieboard2 has been benchmarked at 2GB/s - I'm sure that there are other SoC's which would also work well enough.Quote from: denizcanReally?OK, so it would be 6G (a 1920x1080 frame is 6MB rather than 8 ) for 1000 frames. That, let's remind ourselves, is 16 seconds of video at 60fps. Now the installation that Mike describes has multiple screens - not sure how many but let's say 20. Unless you can send 360MB/s to all nodes simultaneously this means local RAM so you now need 120GB in the whole system for 16 seconds worth of video per screen.Easy, perhaps but not cheap. If you need more than 16 seconds of video available for playback the RAM required starts to go up. 30 seconds and you're at 12G, 60 and you need 24G - that's now not "easy" either.JPEG compression can achieve 20:1 at reasonable quality which means a lot less RAM and you can start to consider getting frames off an SD card although you need one which can do better than 18MB/s. I think I-Frame only H.264 should get better ratios and both should be do-able in an FPGA if that's what Mike wants. Both should also suit a SoC solution. As I said previously if I were building this I'd at least try a Cubieboard2 per display. I don't think a Pi has the horsepower or the RAM really although Mike hasn't said how many seconds video he wants available.

author: nctnico
author_rank: Super Contributor
timestamp: November 06, 2013, 03:06:50 pm
content: The SoC on the Raspberry pi is actually intended for video-over-IP setop boxes. Therefore it has a hefty hardware video decompression unit. The downside is that Broadcom doesn't provide any specifications or support unless you buy >6 digit quantities.

author: grumpydoc
author_rank: Super Contributor
timestamp: November 06, 2013, 03:20:03 pm
content: QuoteI'm aware of that - however it's spec is 1080p @30fps, Mike asked for60fps although subsequently hinted that he was only after 1280x1024. The Mail GPU in the A20 is speced at 3840x1080@30fps (for 3D) so should be able to do 1920x1080 at 60fps with no problems.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: November 06, 2013, 03:46:58 pm
content: To avoid any confusion, there will not be multiple screens (or if there are, they will each have their own controller and locally stored content).Resolution vs.framerate is flexible, but 1024x1280@30fps would be an absolute minimum, but the more options the better. We'd like to drive LVDS panels direct, so this would probably need 60fps refresh from video RAM, though 30fps compression with buffering should work.Latancy is the key thing we need to keep under tight control.And there are quite a few sysem-on-module products based on the Freescale i.MX6 processor that look at least worth spending time investigating, and if they will do the playback they also open the possibility for generative/rendered content.It looks like it has all we need - SATA, reasonable RAM, HDMI, LVDS (even MIPI for tablet-scale displays) and viceo decompression hardware. With a ready-made module, we can add a custom carrier PCB with just the stuff we need without getting into many-layer PCB, length matching DDR3 & all the other voodoo stuff.We probably need to find a Linux (or bare metal0 low-level person to delve deeply enough to evaluate performance, as standard drivers may not do quite what we want, or do it fast enough. (any offers please PM me - there is a development budget)The other aspect is that once the low-level video stuff is done, it needs to be programmable for the behavioural & content aspects by non-elite level, non hardware-oriented programmers.

author: grumpydoc
author_rank: Super Contributor
timestamp: November 07, 2013, 08:04:03 am
content: QuoteAh, all of your examples were multi-screen so I assumed that the current project would be also.QuoteYHM

author: DutchGert
author_rank: Frequent Contributor
timestamp: November 07, 2013, 09:44:23 am
content: What about using Qseven modules and bring out just the interfaces u need?Low-power, powerfull, easy to interface with and fast development.http://www.qseven-standard.org/

author: grumpydoc
author_rank: Super Contributor
timestamp: November 07, 2013, 02:31:21 pm
content: Quote from: mikeselectricstuffI see now where the "60fps" requirement comes from - more that you want to drive the panel at 60Hz rathar than decode video at 60fps.Quote from: nctnicoRe-reading your comment I don't think it's that bad.For one thing everyone (including me) has been thinking in terms of the amount of storage for uncompressed RGB frames which isnotthe most sensible way to do it - you really want to avoid colour space converions in software if you can so you'd keep the decoded frame as YCbCr - owing to chroma subsampling this should be half the size at 3MB per frame.You alsodon'tcopy the data around in RAM - at least not if you can help it - just tell the GPU where the start is and Mike's already said that decoding and caching frames as part of the system start up is acceptable so you only really need to sustain 180MB/s out of RAM for the display.Quote from: mikeselectricstuffCubieboard2 will do that for you. Which Freescale boards(s) have you looked at?Quote from: mikeselectricstuffYou probably don't need to get too close to the metal for the video side - SDL+ffmpeg+appropriate hardware acceleration library should do you. Pretty certain that you can do hardware accelerated ffmpeg on the Freescale. You might need a driver for a touchscreen if that's how you want to interact with it.I presume whan you mean low latency you want the system to respond quickly to user interaction (ideally I'd guess within one frame)Probably H.264 video with a very short GOP (either Iframe only or perhaps a GOP length of 2 or 3 with no B frames) will do for the storage - I suspect youwillbe able to put your video on an SD card and decode on the fly for most or all of the platforms that have been discussed.Quote from: mikeselectricstuffNon-programmers I'd have thought.

author: denizcan
author_rank: Regular Contributor
timestamp: November 07, 2013, 11:08:40 pm
content: You guys always have to be against something..QuoteDid I said "proof of concept"? Let say it is not 60fps, but 30fps, ok we could not manage 1920x1080, make it 1280x720.. What we loose? 50$ of a board. Mike spends much more money to things to take it apart..At most it may lay around to play with.. But we have tested our idea.. Did I said quickly? In a day, one can create a program that plays frames.. The rest is the sensor processing..QuoteI calculated raw 32 bits, you may use 24 bits, YUV etc.. I think you also missed what Mike aimed.. He wants to solve the problem at most 300$ per screen.. So, one board per screen.. $50 board passes this, right?

author: grumpydoc
author_rank: Super Contributor
timestamp: November 07, 2013, 11:23:53 pm
content: QuoteIndeed - I'd go for a cubieboard2 which is about £60 and does all of the things in Mike's list (LVDS, HDMI, SATA, 1G DDR3) so well within the budget (and I think I already said that). I think Mike's looking at freescale solutions which I guess are similar price and capability - I've not played with any of those.Mypoint has been that storing uncompressed frames is going to eat rather more RAM than most SoC solutions have - or is going to result in only having a pretty short duration of video available BUT that it should be easy enough to decompress on the fly on pretty much any of the hardware platforms discussed so far. Note that I haven't said it couldn't be done (if you're prepared to throw quite a lot of RAM at it).Admittedly I got slightly thrown by Mike's original statement that he was after 60fps video when I think he was thinking about driving the panel at 60Hz and also that since all of his original examples were multi-display I thought that this one was going to be multi-display as well.

author: denizcan
author_rank: Regular Contributor
timestamp: November 09, 2013, 12:53:45 am
content: Quotecubieboard2 looks really promising.. my r-pi sample was just food for thought..QuoteI agree on that. From the sample video, even if the decompress time is a problem, I am sure Mike can find a deterministic way to forward decompress and cache.. From his messages, I understand he loves optimization..