Software: Ideas for fast streaming of disk data in Retro Z80 design? - Page 1
Forum section: electronics / general

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 02, 2016, 07:27:09 am
content: HiI am building my Retro Z80 project (see blog link in signature) and am researching how to best hook up file storage. I figured off loading the Z80 CPU as much as possible would be preferable in order to keep application programming as fast and simple as possible. I wouldn't mind putting a micro on the Storage interface hardware to manage details of interacting with an SD card or IDE/PATA HD and do local buffering etc.So instead of the Z80 having to deal with drive sectors and the complexities of a file system, I would like to abstract all that into a stream based API. My first idea is to have a generic OpenStream function that takes an uri that points to the resource. For example: "file://[named storage]/root/directory/file.ext". Using the uri scheme has the advantage that other protocols (like http) can be accessed using the same API. The disadvantage is that the string based nature of an uri probably introduces some overhead compared to a binary interface. However my Z80 design has multiple banks of 64k (1MB max) so that may be worth it. The Stream* that is returned could be compared with the C FILE struct, although physically it will probably be different.The protocol handler for file behind OpenStream will implement the driving the hardware interface to communicate the storage file content. I see three options:DMA - The storage interface hardware requests the Z80 bus and pumps a block of data directly into RAM (MMU RAM blocks are 4k). This option will probably be the fastest but perhaps require extra complexity in the software to manages the blocks?Serial IO - Using I2C or SPI (UART?) to serially communicate the data. Problem here is that the Z80 has no hardware for this. For this to work, it would require extra hardware and probably still be the slowest option.Parallel IO - Use Z80 input and output instruction to communicate with the storage interface hardware at a specific io address. This would be fairly easy to do both in hardware and software.Functionally Streams also have to be seekable (for files anyway) so the interface with the hardware interface must also support that.Finally, the question is if this sound reasonable or are there better ways of doing this? If you have any experience with this, I would love to hear from you. Thanx!

author: hamster_nz
author_rank: Super Contributor
timestamp: May 02, 2016, 07:49:57 am
content: I have only limited experience of this, but it sounds very odd to put an OS level (or even higher) layer of abstraction at the lowest-level of the design.I would have thought that you would want the lowest level to closely reflect the physical attributes and capabilities of the physical device being accessed (maybe block, character or packet, like in Unix land), then build on top of that

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 02, 2016, 08:14:37 am
content: Why?I know that is how it is usually done, but if you want to off load as much processing as possible from the central CPU, you need to go to a higher abstraction level. I want to have just enough control to be able to do all the stream functions without any extra added details.That would mean that low level functions like format, defrag etc would indeed be implemented by the storage interface hardware and be available through a separate API.

author: hamster_nz
author_rank: Super Contributor
timestamp: May 02, 2016, 08:44:25 am
content: Quote from: obiwanjacobi on May 02, 2016, 08:14:37 amEnumerate the API/features you want to support, and then the different resources that you want to access them. They naturally form into groups (with a few oddballs like I2C addressing, and configuration).Open, close, create, seek, read, write, some sort of wait for data, error detection/status, some way to enumerate and organise things into a hierarchy.Maybe have a look at how a high-end SCSI card works, with a command queue that is used to pass requests and data to/from the target. That is a pretty generic 'talk with anything' interfaceHow do you envisage the equivalent of directories working? Do you want to be able to give objects attributes other than just holding data? (I.e will you have a file system at all)

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 02, 2016, 09:34:48 am
content: Yes, determining the final shape of the API is not done yet. I am doing exactly like you've said and writing down the protocols I wish to support and what the common denominators are. I didn't mention it - not to blur the prime objective of this question. Also thinking of streaming audio in this way - to give you an idea of what my thoughts are...The functions/operations I plan to support may be derived from the http set of verbs: get/post/del etc. Not sure yet. Whatever makes most sense in most protocols.I have not yet determined the needs for meta data or header data in any detail, but my initial idea was to be able to open meta-streams. That would allow getting/setting file attributes as well as specifying http-headers if it would ever come to that.Hierarchical information is retrieved on a stream per node basis. So "file://" would return a stream listing the directory just like a dir command. Cool thing here is you could do "file:///*.txt" to return only text files.Specific streams will have helper code for reading and writing. So a set of FileStreamReader functions would help with navigating over the stream content.(I am also thinking of adding extra support for chunk-files that I may be using as a program/executable format.Files like .wav and .mid are chunk files.)

author: stmdude
author_rank: Frequent Contributor
timestamp: May 02, 2016, 09:49:03 am
content: Have a look at what CBM/Commodore did with their disk-drives for VIC20/C64/C128. It seems very similar to what you want to do, so maybe you can get some ideas from there.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 02, 2016, 10:02:51 am
content: Quote from: stmdude on May 02, 2016, 09:49:03 amGreat idea, thanx!Edit: found this:http://c64emulator.111mb.de/c64/docu/IEC_1541_info.pdfVery low level...

author: ale500
author_rank: Frequent Contributor
timestamp: May 03, 2016, 04:20:23 am
content: I think you have here 2 different concepts, that (imho) need to be separated.The first one is to get data from a (any) source quickly. Use DMA, burst modes and so on.A (software, OS) layer that allows you to request/send a variety of content to/from different devices.CBM solved (and like it may OSs) with the IEC bus (sadly, no DMA) and with its kernal. "Device not present" was a great feat. The PC didn't have something like that... but copy to serial or to printer and so on, that was there.I'd not confuse the transport with the kind of data you put on the bus. Define something that (already) works, you do not need to re-invent everything. When you want to test something, the best is if you have some server software on your PC and "serve" content via a RS-232 (at 1MBit for instance). Easy to setup, debug and program. Even testing a new ROM can be done this way...

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 03, 2016, 05:39:57 am
content: Yes, there is the hardware transport and the software API. But I felt that a specific choice on the hardware may have consequences for how to structure the software. As I sort of have an idea how I want the software to look - would that have consequences for the hardware? I understand that you could see these as separate but would that be optimal? My question here is to gather more input for making these decisions.

author: MrSlack
author_rank: Frequent Contributor
timestamp: May 03, 2016, 05:50:50 am
content: Another one. If you're using an MMU you can use a virtual address space and memory mapped files/streams. The pages don't have to be held initially in RAM. With some logic you can demand load them from whatever you are streaming from in 4k blocks if there is a page fault.I don't know how complicated your MMU is going to be though.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 03, 2016, 06:01:49 am
content: I do not have virtual memory in a x86 sense. I do have multiple 64k banks and 20 memory address lines organized in 4k blocks I can put anywhere in active (64k) memory address space. A couple of blocks will need to be fixed (page0 and stack etc), all others can be de/selected.Actually I am currently working out the details of this in software (together with integrating it into the sdcc /z88dk). So I do not have the final details yet, but the idea of DMA transfer would be to use 4k blocks of memory to store (parts of) the file in with the stream abstraction on top of that. When block is almost read in its entirety, a new block can be fetched (async) and reading can continue on the new 4k block. For a short time you need 2 4k blocks, otherwise only one. For other protocols I hope I can use the same mechanism...?

author: MrSlack
author_rank: Frequent Contributor
timestamp: May 03, 2016, 06:35:26 am
content: That would work. The first Unix versions used a similar method of abstracting streams. Look for Lyons Commentary on Unix - there is a lot in that about this sort of stuff.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 03, 2016, 04:58:58 pm
content: Quote from: MrSlack on May 03, 2016, 06:35:26 amYou mean something likethisandthis?

author: MrSlack
author_rank: Frequent Contributor
timestamp: May 03, 2016, 07:09:47 pm
content: That's the ones. Also Operating Systems Design and Implementation by Tanenbaum.

author: ale500
author_rank: Frequent Contributor
timestamp: May 04, 2016, 04:37:22 am
content: QuoteYes, but that is exactly why I suggested using a serial connection for test: abstraction. It adds a layer but keeps your software/API independent of the transport medium being a disk that answers in bulk or a serial that gives you one char every other moon. The unix concept "everything is a file" may not be too far off.

author: Rasz
author_rank: Super Contributor
timestamp: May 04, 2016, 04:28:22 pm
content: sounds like you want network file system

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 06, 2016, 07:54:42 am
content: Quote from: Rasz on May 04, 2016, 04:28:22 pmIn a sense, yes. If you replace the network with an internal bus.

author: legacy
author_rank: Super Contributor
timestamp: May 06, 2016, 04:25:34 pm
content: what about a super simple NFS over a super fast serial @ 1Mbps?you can design it in HDL

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 07, 2016, 08:02:17 am
content: Ok, perhaps the situation is not clear (or I don't get what you're saying).On one side there's the program that is running on the Z80 and on the other side, some kind of smart IO device - in this case a HD or SD card. "Smart" means that it has the brains to elevate the protocol up from the native device it interfaces. I want to extent this idea for other types of IO devices as well - if at all possible.The idea is to exchange data at an optimum rate between the Z80 program and the smart IO device. Because the Z80 is processor and not a controller it has no on-board serial capabilities. What it can natively do is read/write 8 bits from/to memory address space and the IO address space (both 64k).So you see, any serial interface would require extra hardware and translation because the Z80 is a parallel beast, not a serial one. The two best/simplest options that remain are DMA (the smart IO device reads/writes data directly from/to the Z80 memory) or basic IO instruction (we use the Z80 IO address space).As for HDL, I have (more than) a couple of Altera Max II's laying around I want to use for this project to keep it low cost. I already use one to do all the address, IO and other decoding (enabling memory chips etc.) for the Z80 system. I know a little VHDL.I think that DMA would be the most optimal option. The only problem is that previously I only had one 'entity' being able to request the bus - the System Controller (A PSoC5 that takes care of PC related comms incl. downloading initial software. The system has no ROM) - where as now, we would be moving to a scenario with possible multiple 'masters'. But that should not be a problem. It would only require some small changes in the PSoC5 software that currently assumes it is king...So with the hardware transport figured out, the real challenge that remains is the software protocol / API. What would be the optimal interaction between the smart IO device and the Z80 stream functions? Of course I can naively implement this and I am sure I can get it running. But perhaps there are existing algorithms that work better. I have looked at the unix v6 code but I cannot see any magic there. So perhaps there is no magic...?

author: hamster_nz
author_rank: Super Contributor
timestamp: May 07, 2016, 08:40:02 am
content: Hum. Dual ported RAM and aimple signalling would be the least effort.An 4k block of DP RAM, an 8 bit register for ownership, an 8 bit register to indicate CMD/data and an IRQ signal. The 4k block is logically divided into 8 512 byte buffers.To start a transaction the Z80 looks for a block it has ownership of, sets the command bit, then writes the command into the dual port RAM. It then sets the ownership bit to hand over to the external I/O processor.When the command is accepted the I/O processor writes the reply into the buffer, and clears the ownership bit, and pings an IRQ to the Z80.If a data transfer is required (e.g write or read), the Z80 fills the buffer, clears the command bit, and sets the ownership of the block back to the I/O processor. The I/O processor can then use that buffer to pass data back to the Z80.That way you can have upto 8 I/O commands active at once.If you sit the dual-port memory in the correct address window you could also use it to bootstrap the Z80

author: legacy
author_rank: Super Contributor
timestamp: May 07, 2016, 10:22:26 am
content: Quote from: obiwanjacobi on May 07, 2016, 08:02:17 amCode:------------------| super uart0 ===== NFSlike over serial on Cat5 === linux SoC @media{SD,USBHD,...}, fs{Ext2,Ext33,Vfat,...}cpu --- | FIFO super uart1 =====| super uart2 =====-----------------fpga

author: legacy
author_rank: Super Contributor
timestamp: May 07, 2016, 10:23:43 am
content: Quote from: hamster_nz on May 07, 2016, 08:40:02 amthis is a good solution, used in avionics

author: C
author_rank: Super Contributor
timestamp: May 07, 2016, 03:01:09 pm
content: When looking at an mass storage interface, think you need to look at the overhead to get data transparency. A byte in a block of data in memory or on mass storage can have any value. Bytes in the control and status can also have any value.Some hardware interfaces can handle this in hardware.You have hardware based packet interfaces like ethernet, HDLC, SDLC where the hardware defines the starting point of a packet in hardware. A counter value in the packet then keeps the the three(command, status, data) separate.SCSI uses a 9th wire with the 8-bit data buss to do the separation. In addition allows many masters on the same buss.A uart is a very poor choice for mass storage.Look at how many times a block of memory to/from mass storage must be moved.How much software time is needed.TurboDOSA Multiprocessor Operating System for Z-80-based computershttp://www.cpm8680.com/turbodos/TurboDOS 14 Implementers Guide.pdfhttp://maben.homeip.net/static/S100/IMS/software/IMS%20Z80%20TurboDOS%2014%20Implementers%20Guide.pdfThe main limits to TurboDos was keeping compatible with CP/M user programs.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 07, 2016, 03:52:03 pm
content: Using DP RAM is an interesting idea. I am not sure if its the least effort, though - because I already have 4 banks of 64k on my board, but not DP RAM... Also DP RAM is excellent when there is contention on the RAM addresses. Here, using a block based transfer of data, I do not see that contention. It would safe the Z80 from being interrupted by a BUSREQ though... Is that the reason to use it?As for serial, I still don't see how that is 'better'..? It just adds hardware and translations/transformations... Or I am really not getting it.I do not think I want to fix any parameters (packet size) in hardware - I already have a fixed memory-block size of 4k. I can imagine that it may make it even faster if I did, and perhaps I will never need the flexibility, but I just don't want to commit to any value(s) at this point.

author: hamster_nz
author_rank: Super Contributor
timestamp: May 07, 2016, 08:25:19 pm
content: Quite a few have offered ideas, most of which has been discounted.Maybe we should turn the question around.... how do YOU suggest YOU will implement a fast, high-level abstracted I/O interface, on an 8-bit CPU from 40 years ago, without using a serial interface, without using a shared memory space, without having an arbitrary restriction on transfer size, and without having a list of the feature set that you want to support in the first place?WE can then discount them as being unworkable because they will require work,or because it doesn't allow some feature that you did not know you wanted

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: May 08, 2016, 04:12:58 am
content: Quote from: C on May 07, 2016, 03:01:09 pmThere's the rub. What's the point of designing a super-duper DOS if there's no software to run on it? OTOH, why should a system be defined by a crappy OS like CP/M just because it uses a Z80 CPU?So the real question is: just exactly what do you want todowith this super-duper Z80 based system? Perhaps it's all about the challenge, and as soon as you get it working you will lose interest. Nothing wrong with that, but other people might be more interested in something that is actuallyusefulrather than just an engineering exercise.I am have designed an expansion board for theMattel Aquarius- possibly the crappiest home computer ever produced. I ripped out the crappy modulator and replaced it with RGB. My expansion board adds 32k RAM, an AY-3-8910 sound generator with stereo output, and a CH376 USB disk interface. Now I can play the thousands of chip-music files that can be downloaded off the net - onrealhardware rather than in a crappy emulator.The CH376 connects directly to the Z80 data bus, and achieves a transfer rate of about 120kbytes/s. On a machine with only 34K RAM this is blindingly fast, but it could be even faster if the CH376 didn't make the Z80 wait every time it accessed a different sector on the USB drive. So I have been thinking about making a replacement that makes better use of the Z80's limited processing power. One of the things I want to do is play full-motion video and swf style animations on the Aquarius. Why? Partly for the challenge (getting it to do something that back in the day was considered impossible) but also because of the artistic possibilities.But what have my plans for the crappy Aquarius got to with a suped-up Z80 system with oodles of RAM and DMA etc. etc.?Streaming. The CH376 would work much better with the Z80 if I could just open a file as a stream and read/write data on demand, rather than having to put up with the CH376's limitations (max 255 bytes per transfer, have to wait for sectors, max 65535 byte file size). It would also be nice to be able to have more than one stream open at a time, with minimal overhead when switching between them.Why should the Z80 have to worry about disk sectors, file allocation tables etc., when all that can be handled by external hardware? A modern MCU with a bit of RAM should have no trouble talking to a USB stick, SD Card, Ethernet/wireless etc. and buffering the data so the Z80 can get it when it wants rather than having to muck around doing CPU hogging low level stuff.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 08, 2016, 05:51:30 am
content: @Bruce: my thoughts exactly! The whole idea is to off-load the CPU as much as possible. Sounds like a fun project, that extension card.I don't think I have a real goal other than learn from the experience and I love thinking, building and writing (about) this stuff. In the back of my mind, I have this idea of building a simple gaming platform, kids can work with. So I want the hardware to be fast and the software to be simple. I AM using this project to learn other/new stuff as well, like the PSoC5, the Altera Max II and VHDL and PCB design (have made two boards for it thus far).It is not my intention to discount any suggestions that are made here. I am just thinking out loud - that is kind of what you do on a forum, is it not? If you think I am not realizing I want a certain feature, tell me. Thing is that I have a clear picture of what I have and what I want (short term) in my head. You guys don't. It takes too much text and effort (on both sides) to completely communicate that. Also people tent to assume using a retro Z80 also means I want to use all the period correct support chips. I don't.In my mind the 'best' design is using DMA or the DP RAM (if it is to work around the BUSREQ thing - is it?) for the main data and metadata streams. I think I would build a storage board (PCB #3) that has an 40 pin PATA/IDE connector on it and an SD card slot on it. Put a nice micro on there (or a PSoC4?) to make it smart as well as some extra RAM for buffering, some of which may be mapped into the main system banks (DP RAM).Perhaps I am looking for some answers to problems that just aren't there and my naive idea of how to implement this, is actually how everybody has done/would do it...

author: hamster_nz
author_rank: Super Contributor
timestamp: May 08, 2016, 06:44:24 am
content: The dual-ported RAM idea (which could also be implemented by DMA or so other memory sharing techniwue) should was just a simple model to tie Z80 to the I/O processor.However, what is being discussed by Obiwanjacobi and Bruce will most like sound familiar to old-timers here.In the good old days, when RAM was measured in bytes, and computers in tonnes, these techniques where very much used - IBM had intelligent terminals, connected to intelligent cluster controllers, which connected to I/O controllers which connected to their Mainframes. As much works was performed as close to the user as possible, so that the mainframe only had to process pre-formatted, pre-validated transactions from the users. This minimized the CPU requirement on the mainframe.The system was inflexible - your user interaction had to fit in the 'forms-based paradigm', so you couldn't things like screen editors, spreadsheets or do on-the-fly validation of data files. This is where UNIX minicomputers and PCs broke got their big break - the were very interactive.I remember ICL also have intelligent storage, where database-like queries could be offloaded onto the storage controller, and only matching records would be returned...All these techniques have got the way of the dodo as the processing power went up, memory size went up and cost went down. It became cheaper, more efficient and more flexible to have the central CPU handle all the user interaction and I/O directly - if all the resources (CPU, memory, I/O) are centralized then they can be more efficiently used.The only reason they are being reinvented is that :- the memory bandwidth of the Z80 is only a tiny fraction of the bandwidth of an SD card, or 100BaseT Ethernet, a HDD or any USB device- the memory space on the Z80 is tiny, limiting code size and data size- CPU cycles on a Z80 are expensive - with only a few million per second- I/O on the Z80 is expensive, as Z80s are not that good at multitaskingSo a bit of archaeology of old techniques might prove fruitful....

author: C
author_rank: Super Contributor
timestamp: May 08, 2016, 07:55:48 am
content: Quote from: Bruce Abbott on May 08, 2016, 04:12:58 amWould guess that you did not bother the read much about turbodos.If you had you would have seen,local or remote IOlocal or remote mass storagenetworking built inmulti-processormulti-userdistributed systemOS system that could have Z80's in the 100's and be networked is a crappy OSmore specific for Obiwanjacobi it shows what is needed to have a Z80 supported by many CPU's that works.

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: May 08, 2016, 09:42:58 pm
content: Quote from: obiwanjacobi on May 08, 2016, 05:51:30 amFast and simple is good. Getting both can be tricky, but with modern hardware it shouldn't be too hard. The trick is to get it as fast as possiblewithoutadding so much complexity that the project become too expensive or never gets finished.For me, cost is always a factor. I want my designs to be reproducible by others without needing a lot of expensive hardware and software. That means cheap and readily available chips that can be hand-soldered, free or open source development software, small 2 layer pcbs, cheap dev boards from eBay etc.QuoteWill you be sharing your designs? Even if they aren't finished, it could help others like me who are trying to work with the same chips.QuoteDual port RAM is the best for speed and simplicity. Most home computers used a Dual Port RAM circuit for video, as it provided thespeedof non-blocking memory access by the video subsystem combined with thesimplicityof direct access by the CPU. But DP RAM was hard to implement because it needed a lot of logic gates or expensive custom chips. Now we have cheap CPLDs and FPGAs that should make the job a lot easier (only problem is most of them are 3.3V max and/or in BGA packages).DMA seems like a good idea, but has problems. It steals cycles from the CPU, which a lot of the time ends up doing nothing while it waits for the DMA transfer to finish. It increases complexity because the CPU has to set up DMA registers to initiate each transfer, rather than just using simple memory and I/O instructions. Zilog's DMA chip was inefficient because it took 2 cycles to read and write the data (I/O <-> RAM should only need 1 cycle).QuoteGood ideas, but you shouldn't lose sight of the likelyapplicationof that hardware.The Z80 has limited processing power and memory addressing capability, so even a 1GB hard drive is way more storage than you need. And somehow that data has to be loaded onto it. An SD card or USB stick can provide just as much storage space - but with the advantage of easy transfer from a PC. And it needs less power and takes up less less space.Being able to read/write RAM directly without using the CPU can greatly speed up transfers, but whatuseis it? Now we get back to the question - what do you actually want the computer todo? How will this help to make 'a simple gaming platform' that 'kids can work with'? Home computers such as the C64 and MSX answered this with (relatively) powerful but simple graphics, and BASIC. But they were hobbled by slow storage systems, limited memory, and an inefficient interpreter.Just adding fast removable storage to a retro style computer makes a huge difference to its functionality, and makes it much nicer to use. Include banked memory, powerful but simple graphics (multi-color sprites, flip screens, hardware scrolling etc.) a 'real time' BASIC compiler and assembler/debugger - and you have a winner!But don't get carried away. A simple gaming system that kids can work with doesn't need multi user or a fancy DOS. It doesn't need transfer rates so high that the Z80 chokes on the data. It doesn't need 1GB of RAM or a 100GB hard drive. It doesn't need Internet access (you have a PC for that). It just needs to dosimplestuff,fast.

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: May 08, 2016, 09:50:36 pm
content: Quote from: C on May 08, 2016, 07:55:48 amSorry, I stopped reading when you said "compatible with CP/M user programs". I will have a look at TurboDOS - might be something in it worth using!

author: Jr460
author_rank: Regular Contributor
timestamp: May 08, 2016, 10:50:07 pm
content: Sorry If I didn't read closely everything......I saw Z80 fast disk IO, CP/M.... and then all kinds of overly complex ways to do things that were done with a few 74XX chips 40 years.The first CP/M system I touched with 8" single sided single density floppies had a very simple disk IO controller. Write to an IO port the the track, sector and memory address you wanted to to do the transfer to/from. Write a GO/read/write to command port. Poll that command port for a done bit. When it was done, the data was DMA-ed to/from the memory address you wanted to/from your buffer in memory.(if you want to be slicker about it, then wait for an interrupt to signal the IO is done, you will want that if you want to upgrade to MP/M)The real slick thing was the disk card watched the reset line on S100 buss, when that happened, it grabbed the memory for DMA, which prevented the 8080A or Z80 from fetching anything. It reset the drive seeked to track zero sector zero, DMA that first sector into the first 128 bytes of memory. When it let go of the buss, the CPU now had a boot loader in memory and running it.No EPROMs needed, you get the full 64K for memory for the machine, no bank switch of an EPROM overlay, etc.All the fancy stuff is in the CP/M disk routines, getting more than one sector, interleaving, etc. You could pull some of that down into the hardware. For example reading/writting more than one sector per request.

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: May 09, 2016, 06:30:44 am
content: Quote from: Jr460 on May 08, 2016, 10:50:07 pmYes, that's what we are talking about. Different sector interleaves etc. were a major source of incompatibility between systems, and having to handle all that low level stuff bogged down the CPU and wasted memory. We should be able to just open a file and read/write it one byte at a time, seek to any position, load/save memory blocks... all without having to worry about buffers, sectors, file allocation tables etc. Unless you are doing some tricky copy protection or writing a virus there isno reasonto go any lower than that.QuoteBut you had to keep a system disk in the drive or it couldn't start up. And if that disk became corrupted...An EPROM is a lot simpler, faster, and more reliable. The main reason they weren't used more was that they wereexpensive, and you needed an EPROM burner and UV eraser to program them. Now we have cheap high capacity EEPROMs that can be reprogrammed in a few seconds, evenin circuitif you want.Many design decisions made back then would have been different if more modern hardware was available at a reasonable price. Now that wehavedirt cheap modern hardware, it's time to re-evaluate those decisions.

author: BloodyCactus
author_rank: Frequent Contributor
timestamp: May 09, 2016, 02:54:27 pm
content: get a pic or atmega and stick fatfs on it to a sdcard. have z80 just ask the pic/atmel cpu for a block of data from file x or write block of data at offset x.. offload all disk stuff to the pic/atmel... let fatfs do all the work..

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 09, 2016, 03:13:59 pm
content: @hamster_nz: Yes, the abstraction must fit its incarnations. You don't want leaky or incomplete abstractions. The API should be wide enough to fit all the foreseen cases and flexible enough to be extended at places that might be useful. That being said, I have to go deep to design in asm/C again, being used to OO and modern language constructs. I am searching for a simple but clear API structure.@Bruce: Will I make the source Open Source? I don't know. I doubt anybody has any use for it. It will be too complicated for the beginner and too specialized for anyone else. Also I don't think it will be that great a quality (being a hobbyist/novice myself) and don't necessarily want to add the the junk that is already out there. I think I will publish it (with some sort of license), but I will probably not call it open source. I don't wan't to change the tools I am using just because they're not 'open source' enough by some 'undefined' criteria. I don't care much for that sort of hassle.For me keeping it low cost is also a goal. I buy the older stuff that is very cheap (for instance Altera Max II for under $1) and try to make it work, with what I have got. Having some constraints on your design keeps it real and interesting.I am not adding any sort (Z80) DMA controller chip. The PSoC5/System Controller is currently a DMA controller in that it can initiate a DMA memory transfer. The Smart IO Device(s) could also become DMA controllers because they can also initiate a DMA transfer. Obviously care should be taken in managing the system/cpu bus (BUSREQ) but that should be doable (I think). The micro on the Smart IO Device(s) will have to perform the actual transfer.I am currently playing around with the Stream API (writing .h files) and some of its internal workings (writing pseudo code how it communicates with the Smart IO Device). Writing things down always helps for me...

author: obiwanjacobi
author_rank: Super Contributor
timestamp: May 09, 2016, 03:17:04 pm
content: Quote from: BloodyCactus on May 09, 2016, 02:54:27 pmYes, exactly. And it's how the Z80 talks to the PIC/AVR and how that is represented in the software API what I am trying to (get) answer(ed) here...

author: C
author_rank: Super Contributor
timestamp: May 09, 2016, 04:40:37 pm
content: There are at least two sides to some things in the last posts.What is left out can easily be more trouble then what is included.The worst is the hidden blobs of data that you can not find out when using something a different way.QuoteFirst there is a valid need for sector interleave and this does change due to sector use.Hidden Blob data: CP/M's method is very bad here, When using disks from a different system, you have a big unknown, the hidden blob of data that only the original system fully knows.Here the problem starts out from what has been left out of a program!! The lack of reading and understanding the data-sheets.For a soft-sector disk, by leaving out the simple question of what interleave to use in the FORMAT PROGRAM, you gained all the problems this caused. To be clear, One question and very little code caused a huge mess!!!To head some of you off at the start. The very first WD floppy controller chip had full support to format a floppy to any interleave and then read/write the data properly!! The other floppy controller chip common back then also could do this!Leaving out this one question made moving floppy disk between different systems a huge problem. With the question added, the worst case as far as sector interleaving would be slow read/write speed to that specific floppy and coping the floppy to a new disk could cure this.With each change you gain some things and lose some things. It's a good idea to know what you gain and what you loose.Trying to put an eprom in one of my old z80 systems back when it was designed was a huge problem. An eprom was just too dam slow and would have required a bunch of changes and work around's.A change from CP/M2 to CP/M3 breaks some programs while giving new options.It's been reported that Turbodos breaks 5% of CP/M programs. When I looked back then, Most of the broken were replaced by something better, but some were just dead. What is often not mentioned is what the Turbodos system calls added to a user program. A multi-user accounting program existed for Turbodos.fatfs sounds great until you start looking at how many of the old programs will break.One question that needs answered is "Do you want to write all the programs run on the Z80"?OrWould you like to run many of the old CP/M programs in addition to doing your own thing. I say CP/M here as it was one of the few OS's that would work on different hardware. To get compellable with TRS80 requires some hardware for some programs and some require same speed hardware. A lot of other Z80 system are like this an need a hardware and/or speed match.As for tools to make this easer.There is a Z80 emulator for PC that runs Turbodos. This emulator connects Turbodos's network to a port on Ethernet so it can talk to other turbodos systems.Remember that done right, the Z80 sees the user program interface to OS. Very little if any of the OS needs to exist as Z80 code.For CP/M3, anything that CP/M puts in a bank could be on the PSoC5.For Turbodos this becomes easer to do.And you could start out simple running Turbodos on the Z80 and slowly move more and more OS code to be PSoC5 code.Before trying of thinking up your own API, would be good idea to look at Z80 OS API's. A lot of thought was put into some of these.That second turbodos link shows the API needed for trubodos to deal with hardware.

author: BloodyCactus
author_rank: Frequent Contributor
timestamp: May 09, 2016, 05:39:40 pm
content: Quote from: obiwanjacobi on May 09, 2016, 03:17:04 pmwhat I would do, hook the z80 to the pic/avr as rs232, the just pump some byte requests back and forth. easy. all the fatfs code already exists for pic/avr, just add some routines to sit on rs232 and send/rcv. very simple. hook z80 IO port up to do rs232, stick it in your z80 bios some functions to do open/read/write/close calls.

author: C
author_rank: Super Contributor
timestamp: May 09, 2016, 08:30:07 pm
content: Quote from: BloodyCactus on May 09, 2016, 05:39:40 pmTo get to fast reliable data transferred over RS232 which is async 8-bit serial requires more then just putting 8-bit binary data out the port. For mass storage you need command, status & 8-bit data. Using a 9-bit serial mode could give you a chance, but how long can you send/receive a 250Kbit/s second stream.The 250k is the old 5.25 floppy at single density, not very fast even back then.Remember that the old 8" floppy disk at double density was 1Mbit/s.The speed of a pic/avr doing this would in most cases be slower then the old single density floppy disk!

author: rstofer
author_rank: Super Contributor
timestamp: May 09, 2016, 10:54:10 pm
content: I used the Opencores T80 Core (Z80), hung a compact flash on the databus and implemented CP/M 2.x. I used the same core to implement PacMan. I also implemented CP/M 2.x with a 50 MHz Zilog Z80 board and a compact flash - that things screams through MAC and PL/I. I have also used an LPC2106 to grab music files from an NFS server and feed them to an attached MP3 CODEC. I have a bit of experience with these matters.First of all, a disk is a block structured device. That's just the nature of disk storage. As soon as someone says they want to make it into a streaming device, I get confused. The disk drive is always going to be a block device, no matter what. It just turns out that sequential access is only one way to use a disk, random access (read or write) is another. How do you accomplish random access on a stream?The CODEC had a FIFO buffer and all my app did was keep the FIFO full. The file system took care of grabbing blocks of data and since the blocks are large compared to the FIFO and the FIFO uses data slower than I can fetch it via NFS, everything worked out fine. But at no time was the disk drive treated as anything other than a block storage device.Now, if you want to step over a bit, you could have an IO processor (PIC, AVR, whatever) open files and output the data as a stream of bytes and I suppose you could have a DMA channel accept the stream and forward it to a CODEC or whatever but once you get into running an OS, you're going to be right back to a block structured device. Can you have a parallel stream? Sure! But the OS won't want anything to do with it. Before it can run a program, it has to find it in a block structured directory then it needs to follow the block allocation entries to discover the block number for the next sector which it can finally load into the appropriate spot in memory. Complicating this with a stream is going to get really ugly, really soon. And the OS is still going to have to understand the block allocation scheme in order to create new files (where to put them), erase old files (or at least clear their directory entries) and just about anything else.From CP/M, we know that the OS sets up a DMA address and a block number (some permutation of track & sector) for the next IO. These could certainly be registers in an IO controller as long as there was a DMA channel for the transfer. The channel would interrupt upon completion and CP/M could continue. The idea can be expanded to have a linked list of transfer requests along with a call-back address for each. That would allow a multi-tasking OS to run another task while the one requesting a transfer was put into a wait state (waiting for the call-back). This is done all the time but probably not so much in a tiny machine like a 64k Z80. I never spent much time with MP/M so I don't know how they implemented multiple tasks waiting for disk IO.The source code for Unix V6 is readily available if you want to go for the gusto. Just be sure to implement a memory manager.http://v6.cuzuco.com/v6.pdf

author: BloodyCactus
author_rank: Frequent Contributor
timestamp: May 10, 2016, 07:20:40 pm
content: You can also hook a compact flash card directly to the z80 and bitbang it. the ATA layer cf supports still has 8bit mode so you can bitbang it nicely....

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: May 10, 2016, 07:54:43 pm
content: Quote from: rstofer on May 09, 2016, 10:54:10 pmThere is no reason we should have to see that structure in the OS. In fact, we don'twantto see it.QuoteAs soon as I see sectors, interleaves, and FATs I get confused. Why should I need to know anything about that? I just want to:-1. Create, open and close files.2. Read or write a number of sequential bytes in a file.3. seek to a position in a file.4. List, rename, resize and delete files.I might also want use other types of media in a compatible way, eg. use the same code to save text in a file, print it on a printer, send it over a serial port, or display it on the screen.As tohowthese things are done, I don't care. The drive could store data by sector and track, or scatter it all over the disk. It could keep the file names in a FAT, or in a hidden header attached to each file. It could access the entire disk sequentially or write messages on pieces of paper and send them via carrier pigeon - and I wouldn't care so long as it did it quickly and transparently.QuoteA stream is a continuous sequence of bytes with no gaps. That doesn't mean you shouldn't be able to seek to a particular position in the stream to access individual bytes (if the media supports it).QuoteYes, that's what we want.QuoteWhy? Why can't the block structure be hidden in the peripheral where it belongs?QuoteNo, it doesn't. The only reason an OS has to with deal blocks and sectors is that the media device doesn't do it internally.Traditionally, media devices such as floppy and hard drives were very simple, with the minimal hardware required to get bits on and off the disk. Some were so crude that the OS had to control each phase of the stepper motor and encode/decode the MFM data. But those jobs are now done in hardware.The OS also handled conversion from files to blocks. The advantages of this are that the media device doesn't have to know about files, and the OS is free to create whatever file structure it wants. But now we have peripheral chips that are powerful enough to do the job, and we have standardized file systems so the OS doesn't need to create it own format.So let's say you are building a system and you want to put a disk drive on it. You add driver code for the SD Card and aFAT File System, then your OS interfaces to this code via the following functions:-File Accessf_open - Open/Create a filef_close - Close an open filef_read - Read dataf_write - Write dataf_lseek - Move read/write pointer, Expand sizef_truncate - Truncate sizef_sync - Flush cached dataf_forward - Forward data to the streamf_expand - Allocate a contiguous data area to the filef_gets - Read a stringf_putc - Write a characterf_puts - Write a stringf_printf - Write a formatted stringf_tell - Get current read/write pointerf_eof - Test for end-of-filef_size - Get sizef_error - Test for an errorDirectory Accessf_opendir - Open a directoryf_closedir - Close an open directoryf_readdir - Read an directory itemf_findfirst - Open a directory and read first item matchedf_findnext - Read a next item matchedFile/Directory Managementf_stat - Check existence of a file or sub-directoryf_unlink - Remove a file or sub-directoryf_rename - Rename or move a file or sub-directoryf_chmod - Change attribute of a file or sub-directoryf_utime - Change timestamp of a file or sub-directoryf_mkdir - Create a sub-directoryf_chdir - Change current directoryf_chdrive - Change current drivef_getcwd - Retrieve the current directory and driveVolume Managementf_mount - Register/Unregister a work area of a volumef_mkfs - Create a volume on the logical drivef_fdisk - Create logical drives on the physical drivef_getfree - Get total size and free size on the volumef_getlabel - Get volume labelf_setlabel - Set volume labelNote what is missing from this list -anymanipulation of file allocation tables, blocks, sectors etc. All that stuff is hidden inside the FATfs library and device driver. So why can't that low level stuff be offloaded onto the peripheral? Why can't the SD Card be controlled by dedicated hardware which hasits ownFAT file system library, then interfaces to your system via the functions above? As far as the OS is concerned there is no difference.QuoteCP/M is an antiquated OS that was designed around the limited hardware available at the time. There is no reason why we should have to stick to that model today.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 14, 2016, 01:05:30 pm
content: Quote from: Bruce Abbott on May 08, 2016, 09:42:58 pmWell I have published the sources recently. You can find apost on my project here.I also have taken the concept discussed here one step further and am currently in the process of designing (together with Ron) a backbone (bus) that should support smart IO devices as well as other CPU devices (they're the same to the backbone). This would allow me to create a multi processor Z80 system with shared IO (and memory) - how cool is that.We have started with a DP RAM (as suggested here) as a central means of exchanging data, but I have some trouble wrapping my head around the concept. When I read back this thread I see have question marks in every sentence about the DP RAM. The most simple solutions I can think of would not really make the exchange significantly faster (I think). The only advantage is that the exchange -at block level- would be asynchronous, but the sender and the receiver still need to synchronize the blocks themselves, the sender signalling the block was written and is ready to be read and the receiver signalling the current block was read and the next block can be written. I can think up very complex exchange and handshake protocols but the question "will it be faster" will be hard to answer on paper. I don't mind complex, but true beauty is usually in the simple and elegant solutions...

author: C
author_rank: Super Contributor
timestamp: October 14, 2016, 04:32:21 pm
content: Bruce Abbott & obiwanjacobiYou both would gain a lot if you were to read some of TurboDos documentation.First a File control block is a space for the OS to store the information it needs for file access. Even in CPM it is a black box for a user program.Think this through, the is Information needed and if you try to put it in the OS as some point the OS runs out of space, as simple as that. An OS that has a limit on number of open files and wastes that space when less then that limit.By having the user program allocate file control block space from user memory that problem is removed. No wasted space, no limit on files.So in very simple terms, Your smart device has memory limits, this puts a limit on file control blocks. Try to access more files then limit it fails. Where file control blocks in user program does not fail.File control blocks exist in todays operating systems and most are stored in user space for just this reason.The list of functions exist as user calls to OS even in CP/M.In todays programmingf_open - Open/Create a filecreates a file control blockSo simple reading of what Bruce is saying is that his user program is messing with system data breaking the separation of User Space and System space and not using system calls.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 14, 2016, 05:54:07 pm
content: Quote from: C on October 14, 2016, 04:32:21 pmWhich ones? I have scanned the implementation guide and I didn't find it very useful - or am I missing something?The otherlink you gavepoints to a whole bunch of docs.

author: bitslice
author_rank: Frequent Contributor
timestamp: October 14, 2016, 08:18:41 pm
content: Why use a DP memory?data is being blatted in one direction, and new data back again, random access to that data is not required.if you used DMA channel to write to a FIFO then you don't have to worry about leaving a 4K memory hole anywhere, just one I/O address is needed.

author: C
author_rank: Super Contributor
timestamp: October 14, 2016, 08:37:06 pm
content: Ok TurboDos basicsSingle user system on Z80 could do it all.Multi-user, each user gets a Z80.Normal for multi-user system you have a additional Master Z80 to do system work.You can add headless Z80's to system.So you can have a batch processor.A slave can do IO. Was not unusual to have a slave connected to system hard drive. A slave could run network between systems.A CP/M drive could be anywhere, remember networking is built in. Same is true for serial ports & printers.CP/M has limits like number of drives, max drive size. A user could run a program and change all the mapping. When a physical harddrive was larger then the limit, you used a different drive letter.One system I worked on a short time had 16 users on S100. For a large school class you had a stack of these, A network between classes, network between buildings. A floppy drive could be anywhere in the above.Try this, CP/M understands serial ports. You could have one z80 controlling the console of many.So to keep is simple get one Z80 started on s100 and it can do what your psoc is doing. Slaves can be very simple. That one Z80 could do same with a 68000. You can have a system that only has one rom or eprom to start it all. Need to change the boot of slaves is just a simple file change.Some slaves only used 2 k of the system bus address space. Think of 68000 like this 16x 68k = 16x2k while the 68k could have 24 bits of ram.Your Z80 with 1m could show on system buss as 1 meg or 2k.That the implementation guide shows is what is needed to make the above happen. CP/M programs run and a program that taps in to TurboDos system calls can do more. One TurboDos program was Multi-user accounting.Think of the above, It is not many people sharing one CPU like unix but more like a cluster.Note that TurboDos is relocatable object files. A good system builder makes each needed driver as a relocatable object file. It is now very easy to put together what is need for each Z80. The Z80 that drives the printer could have very little user program space, no user and a huge print spooler in ram. While a slave can use additional ram for user's benefit.The user Z80 does not have to be connected to system bus. In this case boot and run a program with something to use as a network connection is all. A modem for example. The nice cheap connection is a Z80 SIO with DMA running in HDLC mode. Think of IBM's SDLC network that spanned the world. HDLC is packets just like Ethernet is packets. Just to be complete SCSI is also packets and can be used as a network.If you look on net you can find a Z80 emulator that runs on a pc and uses TurboDos. Your Z80 could talk to TurboDos Z80 emulator on your pc to a TurboDos Z80 emulator on my pc to my Z80The current weak point is your Z80 to TurboDos Z80 emulator on pc. You need something that looks like packets so that you can separate the 8-bit binary data of Z80 from the command/status bytes. Overhead gets huge when you try using Async serial.Now if you improve your PSOC connections.Above was quick could have errors, so ask if something does not make sense.

author: C
author_rank: Super Contributor
timestamp: October 14, 2016, 09:03:12 pm
content: Quote from: bitslice on October 14, 2016, 08:18:41 pmMemory makes everything cheaper and easer. You have a bunch of CPU(s) connected to a system buss. On power up all but one are held in reset. The one then loads this ram with boot code and releases reset. For a Z80 this is address 0. Other types of CPU can have a different boot address and still work. No Roms to update.After Boot you have a shared ram area to use for communications between CPU'sThere are many options to do this.System buss access to this board becomes a DMA into this board.Dual port ram is faster. local CPU is not effected as much, each only have to wait it the other was using the ram.Now bitslice how many channels can can your idea support. And the FIFO needs to be 9-bits wide or have something that says start of packet that does not get messed up when sending 8-bit data.

author: C
author_rank: Super Contributor
timestamp: October 14, 2016, 09:33:26 pm
content: obiwanjacobiDP RAMThis simple example may help.Old computers used DP RAM for the video display. The video controller was pushing limits and when it needed data from ram for display and could not get it a glitch happened to video display. Even with this critical need there were times when video ram was not needed. Some time during horizontal retrace and bigger time during vertical retrace. A lot of poor computers did not check or have ability to check for horizontal retrace or vertical retrace and caused a glitch on the display.If and this is a big IF back then, you could get memory 2x faster then display controller needed, cpu could access between display controller reads.Think of what simple DP RAM is. A 2->1 data selector(74157) on address,control & data in. A 1->2 on data out. Some used tristate to do this. The remaining part is timing and control.Think of two Z80 accessing this memory. One gets it while other gets hit with a wait state. The waiting Z80 lines to or from this memory is static.On access switch change you haveSwitch Input selector changeDelay mreq until address buss is stable after input selector.Delay WR if it is a write so data buss is stable.Most also delay RD so address is stable before RDIf RD change output mux or direction.Remove wait at proper time after delayed Mreq.The Time delays were often created in steps of two inverters.Better choice is to use a delay line on switch only delaying on switch change and leaving switch at last access.A ram card plugged into a system buss is multi-port due to how the many CPUs get access to the system buss. Need to remember that the Z80 does not know it will be accessing the system buss. The address decode says system buss needed and creates a wait to Z80. When it gets the go ahead, the go ahead signal creates proper timing for the bus drivers to system buss and removes the wait.So system bus is like dual port ram to local Z80.A triple buss system is easy if you plan ahead.Talking of an on card buss, local expansion connector and system buss. You need an "I AM Addressed" signal. Your chip select signals get fed back to Z80. The path back to Z80 determines what buffers to enable and what direction.For on card memory this is a simple if RD & "I AM Addressed" change direction of data bus driver to on card memory. It's this simple for local expansion connector also. You then get if not on card or local expansion it must be system bus so start making a request for it's use.From system bus you have a system buss address decoder to say this card is addressed. A system bus memory cycle becomes access to dual port ram or a DMA request for local CPU.

author: bitslice
author_rank: Frequent Contributor
timestamp: October 14, 2016, 09:36:42 pm
content: Quote from: C on October 14, 2016, 09:03:12 pmSorry you lost me a bitI thought we were talking about a single Z80 offloading its disk handling?QuoteWhy would you need more than one channel?Z80 needs to store a block of data,it DMA's this to a FIFO, along with a file name and number of blocks to expectwhatever is on the other side of this FIFO notices a transfer taking place and works out where it's going to put this data.repeat block transfers until done. The Z80 doesn't need to know where the data goes.A request for a file, is the process in reverse with a 2nd FIFO

author: C
author_rank: Super Contributor
timestamp: October 14, 2016, 10:13:53 pm
content: Floppy disk interleavesVery simple, it is a problem created by the system guys.When you format a floppy disk you need to ask the Question,What interleave for this floppy disk.Many systems did this or always used 1 with out operating system changing it.All but hard sectored floppy controllers will read the proper sector.The interleave can be different on a track level.Was not even a big change in the format program. A change from using last sector +1 to using this same variable to access an array containing sector order wanted & the question to build or use proper array.

author: C
author_rank: Super Contributor
timestamp: October 15, 2016, 01:29:38 am
content: Quote from: bitslice on October 14, 2016, 09:36:42 pmAt the simple level what you say sounds good.A request for a file, is the process in reverse with a 2nd FIFONote this has to use both fifo to function.Note that you have a point to point solution.With out channels you have to send more information over FIFOs. At some point both ends of link will need to start sending commands and data. For example, Floppy removed.First to start two CPU's booth need a boot program. In the past you had a choice of ROM or EPROM. To change, Remove and replace.With new choices it is a little easer but still a mess.With shared ram at boot address for all but first CPU, one rom.Shared ram on second CPU removes need for ROM during boot and can be used for anything after boot. The other way is first CPU being able to access Second CPU's memory to load boot program.To work with a remote floppy for example you need,what disktrack numbersector numbercommandEach a byte in size.then send data or receive data.Change this to a hard driveTrack and sector numbers need to get larger or you use logical sector number which is very large.The point here is that the command data changes size.You also need to get back status. So is the contents of second fifo status or data?with 8-bit fifo you need to add a command length field so other end knows how long the command is. Good to do all the time.You have 8-bits used in the data. You need to identify what is command/status from data. You need 9-bits or 8-bit packets to keep this separation.Using 9-bit data through your two fifo's will work if that 9th bit is on for command/status and off for data. But you also have to worry about glitches and errors. You also need to insure that fifo over run is not possible.Now TRSDOS would let a user program change the size of file data block from 1 byte up to 256 bytes. TRSDOS would handle problem if the data effected two sectors on disk.In not working at low level above and using files, same basic information is needed just the names change. Keep in mind that for CP/M the file name is 13 bytes long. Most systems reduce this to an open file #. This change would mean that both ends need to keep the file name to look up the file number. And to speed access the end connected to disk drive needs information for where file is on the drive.If you are using DMA then the fifos are not neededWhat you are proposing is a very crippled version of parallel SCSI but with problems.With just two output ports and two input ports you have SCSI and all the problems are curred. You also have a network in the process and have 8 devices/hosts following initial rules of SCSI.A file control block was use back then and is used today for many reasons. The new name for this is file handles but you need to remember that memory management is also here supporting the changes from a file control block to this new name.part is explained inQuote from: C on October 14, 2016, 04:32:21 pmbut wikipedia is better.CP/M,TurboDos could be better, but at what cost?More needs more memory and or more disk accesses.More for CP/M2 = less user memory.CP/M3 & TurboDos could both use banked memory to cure this less user space problem when more ram existed.TRSDOS swapped parts of OS from disk to ram. CP/M being earlier did not have a space this was possible for all systems.Now my programs used more then one drive at a time and had more then one file open at a time.My 1980 Z80 has 8 floppies possible and unlimited number of hard drives. The file open limit was how much user space was need per file, Not some limit in operating system that wasted space when not used.A smart drive is a bad idea. Some have found even a Smart RAID controller is a bad idea. From the first PC to todays PC drive limits have changed many times and how to talk to the drive. Some have found out their smart raid controller has a 2g drive limit.A smart drive that understands files is a FAIL. CP/M3 and TurboDos both added date stamps. This change would not be possible with a smart drive only knowing CP/M2. It could also not work with other file systems.Stopping here. Ask questions and tell me errors.

author: bitslice
author_rank: Frequent Contributor
timestamp: October 15, 2016, 02:34:58 am
content: Quote from: C on October 15, 2016, 01:29:38 amThe OP mentioned SD card or hard drive, I'm assuming this will be the sole storage device and essentially infinite in capacity. As such I can't see the need for the storage subsystem to report its status?QuoteA glitch would indicate a hardware failure, probably something for an NMI to handle?The storage is going to be faster than a Z80 DMA, I think even floppies outpaced it. If both systems are comparable then the FIFO only needs to be a minimum of 256 bytes (DMA burst block), but 16Kb FIFOs are pennies anyway.QuoteDMA only works in 64K blocks, it seems easier for the memory mapper to deal with one segment of 1Mb and a single I/O port, than manage a DMA between two different segments.If the goal is a CPM system, then yes you'd need to deal with a bunch of dependencies and multiple channels, I was thinking more of what would be an ideal link in a multiprocessor system.

author: C
author_rank: Super Contributor
timestamp: October 15, 2016, 03:11:10 am
content: QuoteAll kinds of status needs to be sent back or user program is blind.SD write error, non existing file, File already existing when opened for output, out of space,....QuoteThen huge fail.Look at the error list of TurboDos it is multiprocessor and compare to CP/M2 or CP/M3 both single processor. Multi-processor & Multi-User both add need to report a bunch of errors.No errors/status back = lost data & poor programs.From every thing I ever heard MP/M was a huge fail, just so you know.EditTurboDos does what obiwanjacobi asked for in first post.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 15, 2016, 06:54:00 am
content: Wow!Lets go back to the basic question: what does a DP RAM bring to a system bus in terms of data exchange. Just so I can understand this.Assume a system bus with multiple smart devices, all of which have some sort of processing on board. Could be smart storage devices or multiple cpu's etc. Doesn't matter.I (think I) understand the DP RAM function. Two ports on the same memory matrix with semaphore control against address collisions. I also understand its value in the video memory example outlined by C. There it makes both sides (the cpu and 'gpu') faster because the granularity of the contention has been reduced to one address instead of the entire video ram. So I thought that DP RAM is 'only' good in scenarios where there is some form of (high) contention over the shared RAM resource.But what form of contention is there in this bus scenario? I assume a data-block exchange mechanism of size-x where data larger than size-x is broken up in parts (of size-x) and exchanged block by block until all data has been transferred. DP RAM size is size-x times the number of channels you wish to support - lets assume only one - to keep it simple.Is the DP RAM intended to make it possible to read and write simultaneously 'inside' a data-block? That seems a lot of trouble for relatively little gain.I can only see that using a DP RAM will make the data-block reading and writing asynchronous with respect to each other. The reader doesn't have to be 'ready' in order for the writer to write its data-block. The writer must wait though, until the reader has finished reading the current data block.

author: C
author_rank: Super Contributor
timestamp: October 15, 2016, 09:08:30 am
content: DP ramIf you buy a DP Ram chip, it looks like two static rams.Each storage cell(byte) is same between two rams. You are talking a newer thing and costly. For most, contentions are at storage byte level and mostly hidden.What I described is normal ram that acts like DP Ram. Was in use in the 70's and later. Contentions happen at total size of DP Ram created this way.Think of first come first served to handle address collisions. At some point in time PSOC & Z80 will try to access DP ram.Think of Your PSOC as the master. For first part here think of PSOC being system buss.Your Z80 as a slave.If you run a Z80 program in DP ram you are using all DP Ram time.Move your Z80 program in other ram and use of DP ram time drops a lot.Zero unless you are reading or writing data to DP ram.Might think of the DP ram as post office boxes.PSOC could be the mail man moving contents of a box to a box on a different board.In this case your Z80 or other board do not need access to system bus(PSOC bus)Now if you think it through your Z80 needs to interrupt the PSOC so it will know to look at a mail box. The PSOC needs to interrupt the Z80 also.You use an in box and out box to coordinate needs. You might make two boxes boxes into circular buffers(console in & console out). Z80 puts data in console out and puts note in out box (check console out box). And interrupts PSOC.Each box has a purpose.Say for example Other Cpu1 has a floppy drive connected.PSOC moves command data from Z80 box to proper Cpu1 box.If Z80 is writting to disk then PSOC moves write data to Cpu1 data in box.Here you have PSOC doing a lot of data moving but very little thinking(processing). PSOC could cheat an pass the buck and have DMA do part of this.Only time address collisions happen is when the other guy got to DP memory first and is still in memory cycle.Note that Mail boxes here are defined by both PSOC & Z80 saying what the arrangement is. Static and dynamic boxes are possible. Size limit is what will fit in DP ram.The block buffers is just a type of mail box.Only one (PSOC or Z80) gets access to DP Ram for a memory cycle. Two asking at same time, one gets it, one delayed a memory cycle.In the video ram example contention happens at DP Ram size. Any Video controller access to DP Ram blocks any other access.Might think of data in DP-ram as separation needed for interrupt data from main program. The Z80 can not do a read-modify-write memory cycle. This is needed to share one thing. You can do one puts data in and other takes data out.So far the Z80 did not access the system buss. The PSOC only needs to talk with DP-RAM on each board. Two boards in PSOC address space is enough if you can map the two spaces to all boards.Your Z80 could swap DP-ram for memory that is at address 0 on reset. PSOC would load boot program just like it does now.All the other smart boards could look just like your Z80. Min would be some DP-RAM, a processor and memory.If the processor is a Z80, with turbodos it is simple. Other processors is possible with more work. You could add a PSOC or arm M?.If you can create a turbodos driver to do the talking with not Z80 CPU, Turbodos will do most of the work.This is very simple and basic, many improvements are possible.When you were looking at my Video DP-ram example, did you think of shadow ram?All reads to video DP-ram come from shadow ram at same address. While a Write to this address range happens to shadow ram & DP-Ram at same time. Contention with video DP-ram is reduced to writes.Questions?

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: October 15, 2016, 03:42:10 pm
content: Quote from: obiwanjacobi on October 15, 2016, 06:54:00 amFor a file system? Not much. You can't use it to read and write whole files directly unless the entire RAM space is dual port. So you have a block of DP RAM that works as a buffer, but the Z80 still has to copy data between that buffer and main memory, with all the usual overheads (slow memcopy, having to break the file up into chunks etc.). However if you are running a disk operating system that already does this then DP may provide some improvement over DMA.QuoteIt can also be used to share data between 2 devices which are operating asynchronously.QuoteYes. Both the writer and reader can access the RAM asynchronously at full speedbutthere must still be some synchronized communication between them (except for things like video RAM, where the CPU can write image data at any time and the video controller simply displays whatever is there at the time).The advantage of DP is that the buses don't have be synchronized, so each device can work independently at its own pace. By having 2 or more buffers in DP RAM the writer can be updating one buffer while the reader is accessing another, then they are only slowed down when one is faster than the other. Of course that means throughput cannot be faster than the slowest device, unlike DMA which (if properly implemented) can go as fast as the bus.But this is not much different to having the buffer RAM inside the faster device. Hard drives are able to read or write disk sectors at the same time as they are transferring data between their buffers and the host. A properly implemented SD Card or USB memory interface would do the same - and could also work with streaming data at a higher level, so the host doesn't have to deal with low level stuff like sectors and file allocation tables.So why would you use DP RAM for a disk device?1. Hardware design may be simpler than DMA or I/O port.2. Disk device doesn't need to have internal double-buffering.3. When file system enforces data transfer at sector/block level.and whywouldn'tyou use DP RAM?1. It's expensive.2. Can only be used between 2 devices.3. Enforces data transfer at sector/block level.4. Not much speed improvement when the host has a slow inefficient CPU.

author: bitslice
author_rank: Frequent Contributor
timestamp: October 15, 2016, 05:29:49 pm
content: Quote from: C on October 15, 2016, 03:11:10 amWe are at opposite ends on this, I feel you are pushing TurboDos as a sole solution and making comparisons with other ideas only in relation to TurboDos's requirements.Streaming data designs never use DP ram, they use FIFO, just as AD converters or video buffers or any other type of high speed point to point data connection uses FIFO.It's a fixed linear stream of bytes, why would it ever need defined XY addresses for each byte? I don't write letters on squared graph paper and let the post office choose in which order to read each character, I write one long stream of words which then arrives as one long stream of words and is read as one long stream of words.It's just conceptually simpler, cheaper and in terms of overhead, faster.DP does have a place in multiprocessor systems, just not as part of a storage subsystem.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 15, 2016, 05:55:07 pm
content: I'm sorry C, but it's not getting any clearer. I don't recognize anything in your post that explains why DP RAM would be 'better' or worse for that matter. I am sure it's me. I really appreciate the effort you put into answering. Thanks for that.Thanks Bruce, that sort of sums up how I understand it. The new insight (for me) is having multiple blocks for the same exchange in the DP RAM, that way I can see where it could be faster -or perhaps less blocking is better term.As for 'streaming': that word can be explained at different levels in different ways. Is a serial connection streaming? - streaming bits over the wire. You could have a streaming interface at byte level - one byte at a time (in a FIFO). You could also see it as a continuous supply of data blocks that provides a software driver with data and the 'stream' exists in software - the client of that driver would not know how that stream came about, it would just work with the API reading (or writing) one byte at a time or in small blocks of data. I am considering all three kinds of streaming in my bus design, however this question was specifically targeted at the data-block level of moving data - and using or not using a DP RAM as the tool for the data exchange.Also, the storage system mentioned is used as an example. The bus design I am working on now is meant as a generic, general purpose 'backplane' to interface any device that wants to talk to any other device, be it two CPUs or a CPU to a Storage device - it doesn't matter to the bus, as long as you play by the (bus) rules.Quote from: bitslice on October 15, 2016, 02:34:58 amI did a quick scan on ebay and could not find any cheap FIFO memory. Care to help me with some keywords to search for?

author: C
author_rank: Super Contributor
timestamp: October 15, 2016, 07:07:29 pm
content: OkAdc on one board to dac on second board. Think it can be called a stream.In ram you want a circulatory buffer. Each end has a pointer to data.read pointer, read/write data, inc pointer, check for wraparound, save pointer.Fifocheck status, read/write port.Fifo could be lower lower overhead. You have to have a fifo between the two systems.Each time you add a Adc/Dac pair you are adding a fifo or changing how you encode data in, decode data out.IT is a fail if the two ends are not close, the endpoints needs to change or you just need more.obiwanjacobiKnow anything about pascal? Might think a step up from C. Some Pascal compilers, compiles to PCode. The huge program is reduced to these steps. You have a language like this needs these simple steps. Change the language, steps could change( Java for example )Using above paragraph. What you see in the TurboDos implementation guide is the PCode needed to1. maintain compatibility with CP/M2. allow multi-Users3. allow one system to act as a source/destination for things CP.M provides a user program on a different system.4. Network was needed for #3, make network available via added system calls.5. some systems could not have some nice to have features do to lack of memory space. Let other systems supply this for limited systems.6. keep memory use low.There are more.So TurboDos is a working foundation. Use it as is or use it as difference to do your own thing.If you look in TurboDos implementation guide you see what is needed for console with other sections for different areas. You might not of noticed that one driver can be used in many different different areas. The OS just uses a different number of entry points.When Ram is on system bus you have _P-ram.Dma by it's actions is creating DP-Ram.The nice thing about Shared ram is that it can change.In C create a data structure that is safe to use between two interrupts.Put the structure in shared ram and move code for one interrupt to a different system.obiwanjacobi does above paragraph work for you to understand shared ram, DP-Ram?

author: bitslice
author_rank: Frequent Contributor
timestamp: October 16, 2016, 02:51:37 am
content: Quote from: obiwanjacobi on October 15, 2016, 05:55:07 pmAh, then you might like to consider a conveyor mailbox.Basically a wide byte packet transfer, consisting of a destination address, a few data bytes and some control bits, this gets passed along a backplane to every potential end point. Each point checks if the address matches and then copies the data if so.Drive this thing fast enough and it matches the byte write speed of a Z80, so every node appears as essentially a P2P connection.I was used in a 256 x Z80 processor AI machine, ZMOBI can't find it online now, but I have the PDF that explains the design, I could email itI'm facing the same design question for my project, I was thinking of something like a round robin 4 channel backplane link, with maybe a 16x16 serial switch matrix as command channels.QuoteTry "IDT FIFO", "Cypress FIFO"supply comes and goes.I last usedhttp://www.ebay.co.uk/usr/exdwhhe has some cheap 36bit wide ones

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 16, 2016, 07:07:41 am
content: The 'conveyor mailbox' sounds like a broadcast scheme. My principle focus for the bus design was point to point and multiple-exchange: multiple device pairs can have an exchange active at the same time / one device might be involved in multiple exchanges at the same time. A broadcast is nice for event notification, but it causes problems when multiple devices are responding at the same time.At the same time I am designing this, I am thinking through a protocol that will run the bus. It's basically a message based protocol that controls how data-exchanges are setup. Each message has a fixed header that can be interpreted by the bus to make sure the messages are routed properly.Quote from: bitslice on October 16, 2016, 02:51:37 amYes, please do (obiwanjacobi at hotmail dot com), thank you.Quote from: bitslice on October 16, 2016, 02:51:37 amIs each channel in that 4-channel backplane, serial or parallel? (if parallel that would mean 4 separate buses)Also why would you need a matrix, would a serial bus not be more efficient?And what kind of serial (protocol) are you thinking of?I think I have sort of the same idea: have a command channel (serial) and a data channel (parallel/FIFO) and a DMA channel (DP RAM) where the DP RAM can be mapped into the native address space of the device (if so wired by the device). I am 'struggling' with multi-channel support on the parallel buses. I do not want to create a lot of separate buses that need to be muxed on/off the devices and have their own DP RAM etc. But sharing the bus (even if assigning unique regions in DP RAM for each channel) would bring contention on that bus, slowing the data exchange down (a bit). I don't mind throwing more hardware at the problem but its hard to guesstimate if that is proportional to the gain.

author: bitslice
author_rank: Frequent Contributor
timestamp: October 16, 2016, 02:49:51 pm
content: Quote from: obiwanjacobi on October 16, 2016, 07:07:41 amWhen I first read it I thought 'meh' too.But their design seems pretty flexible, you can send data to everyone, one node, nodes that talk to a defined source, nodes matching a pattern or a category etc.Basically 256 postvans going round in a circle at 20Mhz, only one of those is your personal postvan, so when yours comes around you drop off the mail you want to send, and by the time you've got the next bit of data ready to write (in 5us), your personal postvan is back outside your door. Meanwhile all the other postvans are independently dropping off mail at your node as required.It seemed a good way to get 256 processors to all talk simultaneously to each other, without blocking any other node.So it's not really a broadcast in which every node stops what its doing to respond, all node comms just happens in the background, and data is moved back and forth completely transparently.The downside is the node comms hardware appeared to require a physical slot, as their node card was full of a Z80A, 8 bit multiplier, serial and a FPU (probably AMD9511?), a small cable then linked the two cards.I was a bit narked when I found out about this ZMOB, because it was pretty much the same as my ideaThere's never anything new under the sun...QuoteDoneQuoteI have a couple of DIN4162 backplanes to use, and 16+ Z80 nodes to link. I was thinking of 4 parallel buses on the backplane, each Z80 node has a DMA fed 8 bit FIFO, one of the 4 channels will then decide to empty a node's waiting output FIFO whenever a channel becomes available (still wondering how to arbitrate that intelligently). Basically fire and forget as far as the node's concerned.This is I guess an array processor, the bulk of the data is going one way, all nodes are asynchronous.Data starts in a DP ram(s) and ends up in a second DP ram.The serial matrix (via signal switch IC's and Z80 SIO's) is just in case small amounts of data have to be passed sideways, or some 'node idle' notification needs to be broadcast. Signal switch 8x8 matrix IC's are cheap and it makes the hardware simpler. Not sure about the arbitration to signal a clear-to-send channel yet.I didn't want to use a modern serial bus as I was a bit wary of speed issues and glitches, I can clock 8 bit data a bit slower. And there is no retro version of I2C.I don't have a protocol in mind as this was going to be a single task machine, and I really don't rate my ability to write parallel code to make it work as a true multiprocessor.QuoteAre you working with one Z80 and the other nodes are (intelligent) peripherals?The major problem with DP is that you can't read the same location that someone is writing to, so that requires fiddly mailbox arbitration - which then either disallows blind burst DMA into that DP, or requires you to wait state the DMA chip (or processor) if that simultaneous access is occurring.It's possibly easier to get a second processor to manage memory for you? just request a block and it dumps it in a block that's uniquely available to you. You can then access gigabytes rather than whatever a MMU limitations are.Or have multiple DP's available and agree on which ones to use for the current transaction

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 16, 2016, 05:26:13 pm
content: Quote from: bitslice on October 16, 2016, 02:49:51 pmBut doesn't that mean that most of the time these postvans are driving around empty taking up valuable 'road' bandwidth? (roads? where we're going we don't need 'roads')I was thinking of only granting a device access to the data/dma bus when there is an actual exchange, which has to be requested with the bus controller (the brains that control all bus matters).Quote from: bitslice on October 16, 2016, 02:49:51 pmYeah, happens to me too sometimes. I had this huge brain wave what later turned out to be 'bit-slicing' and was only 20 years old. I still think my idea is pretty slick thoughQuote from: bitslice on October 16, 2016, 02:49:51 pmSo cool!Quote from: bitslice on October 16, 2016, 02:49:51 pmWhen you talk about background processing (emptying out a device's FIFO queue) you need to talk priorities. File transfers may be higher prio than something else, but lower prio than video output etc.Quote from: bitslice on October 16, 2016, 02:49:51 pmWhy not share the same DP RAM?That is what I was thinking. If you have to DMA the data around an extra time that only wastes cycles. I was thinking of letting the writing-device write directly into the DP the reading-device is reading. I am still pondering on having a central bus-owned (big) DP RAM or have each device have their own (smaller) DP - and whether then to use the reading-side or writing-side DP RAM... (its getting complex)Quote from: bitslice on October 16, 2016, 02:49:51 pmStill, you need to make sure two devices do not target the same device at the same time. I (still) think the same can be made (cheaper) using a bus architecture.Quote from: bitslice on October 16, 2016, 02:49:51 pmI am not targeting a vintage correct setup. So I will use whatever is cheapest and does the job. But I'm having trouble finding a standard protocol that will do buses, broadcast and multi master... (I2C seems like the best option, but is not the fastest. I may have to look into CAN).Quote from: bitslice on October 16, 2016, 02:49:51 pmInitially yes. But when I ordered my PCBs for my Zalt Z80 computer, I got 12 PCBs. I have constructed one at this time. So what to do with all the others...Well.... So it may grow into a multi CPU platform. I would also like to allow other CPUs onto the bus, so I don't want a design that hangs on some unique Z80 feature (like WAIT).Quote from: bitslice on October 16, 2016, 02:49:51 pmI didn't know that. I thought the arbitration/locking was done inside the chip. But that doesn't matter anyway. You can't allow a data block that is being written to be read at the same time anyway, because access is asynchronous - you can't be sure the writer is faster than the reader.Quote from: bitslice on October 16, 2016, 02:49:51 pmMy Zalt CPU board has a MMU (max 1MB local memory) and I plan to make a shared RAM device for on the bus. My bus protocol works with services, so that would be a volatile storage service.The bus itself will have a DP RAM that will be partitioned into channels and further partitioned into block-buffers, so each data exchange (session) has multiple buffers to work with, which allows for faster transfer (I think) -or- each device will have a DP RAM - I'm not sure yet.

author: C
author_rank: Super Contributor
timestamp: October 16, 2016, 07:37:49 pm
content: A memory bus is simple.Good design practice says you use buffers on & off a board.Forget about fancy thinking an start simple and look at some numbers.Back in the 80's where fast memory cost $ you could easily find that a 6Mhz Z80 system was slower then a lesser speed system. Think you will find it is still true at some speed.What most forget is bus length costs time and drivers have limits.A lot of cures happen when you add bus extenders which are simple and one way at a small time cost. And bus repeaters that connect system buses and are two way.Most problems can be minimized.With the correct parts it can scale huge.Think of a processor Card with System bus connector and a local extension connector.A couple different versions of ram cards.A. system bus memory.B. Local bus memory.you are just talking two different connectors here, a buffer ic location change.C. system bus & Local bus memory. Here you just put in both sets of buffers and address comparators, and added some simple time & sequencing.From system buss _P-ram just like A above with added Local P.A Local bus to system bus card.Stop and look at all the ways above can be connected and how processor out and back times change with different paths.One last card a system bus to system bus card set. could be extra connector on Local bus to system bus card.When you look at the times involved and simple to do parts, more gets you very little and often at a big cost.The need for distance created the need for SCSI and other buses like this.Now go back an look at how some can be improved.68010 was built for many CPU's on a bus. Has foundation for hardware and the software. Uses vector interrupts like a Z80. Wants one address bus connected to two separate 8-bit data buses. Can write to one or both buses as needed.This makes looking at the VME bus for foundation ideas. Could be better, easy to make a lot worse.Now the change to above memory cards is really small if you use two cards.One change is A0 becoming part of memory select.For Z80 this means that the system bus to card or local bus to card is half the time. contention between the two buses is half as great and all you changed is to use two cards with A0 in memory select.To be complete where the data buffer ic is installed changes and the RD & WR input pins move. When buffer moves Z80 needs to know which width of memory and use one of two buffers, RW & WR lines.For max speed Z80 you have a choice of clock stretch and/or wait states when you access slow stuff.Should note that A z80 has address stable@10 Mhz 5 ns before MREQ@20 Mhz -15 ns before MREQ, MREQ is saying address is valid before it is by 15 ns

author: bitslice
author_rank: Frequent Contributor
timestamp: October 16, 2016, 09:42:26 pm
content: Quote from: obiwanjacobi on October 16, 2016, 05:26:13 pmheh B2tFYep, the bandwidth is divided into 256 mail slots - but imagine if that bandwidth was 100Mbs and all it had to do was service a couple of slow Z80's - those mailvans would be whizzing past millions of times with nothing to pick up, because the Z80 was still slowly loading the A register with the next byte. Like a keyboard decoder waiting millions of cycles for the speed typist to press the next key.I haven't looked at the timings but they reckon that running the bus at 20Mhz gave them enough time to outpace all the Z80s trying to write to it.Now I don't know if this would keep up with a 4Mhz DMA chip, but then we've both got far few processors to service, plus you can make each mail slot many bytes wide (they used 48 bits I think).QuoteAll our modern bus protocols are built like this because we assume the processor is faster than the bus, but in our case the Z80 is much slower than a backplane, so I reckon it's probably possible to interleave multiple transfers.Quote from: bitslice on October 16, 2016, 02:49:51 pmI saw a 4bit ALU in some Degree students project and thought wow, you can build your own computers!(just at the point that technology became obsolete...)QuoteMy nodes are all doing the same processing task on different bits of data, but I don't know if one processor is going to consistently going to finish its jobs first, maybe I need a supervisor to tweak the workloads more randomly.Everything has the same priority, they just don't need to stumble over each other.QuoteOne DP was going to hold original XYZ point data, and the final DP was going to hold a 3D transform of that original model for final display.Basically, can a bunch of retro Z80's run the Elite game really fast if you throw enough FLOPs at itQuoteIf you make the nodes co-ordinate where the data is being sent, then they would need to know everything about the status of the machine. If however you let the peripherals manage all that, then they already know the node they are talking to, and they already know where to send it (because they are say, the video handler.)Nodes = powerful and dumb, peripherals = slow but smart.If the bus is flexible enough then RAM is a peripheral too, and you could have multiple variations of that peripheral.Actually maybe a truly general purpose node maybe wouldn't have "direct access to a bus", it would just have an input and an output, something else would manage those I/O connections transparently.QuoteI could just increase the matrix size to 16x16x4 and include the hardware control signals, RTS/CTSQuoteYup, every paper on array processors goes way over my head, because they are all Maths PhD's pimping their bleeding edge theory on computation and their theoretical bus hardware is a data centre full of Spartan 7's.QuoteThat thought is exactly what prompted me - "hey, all these boards are going to be the same thing..."QuoteYeh I thought about all the BUSACK arbitration etc, but reckoned that wasn't going to scale above two CPUs - it would be like a field full of old men trying to kick the same football around.QuoteYes it tell you about what's happening, but it still needs your hardware to tiptoe around it.Not a problem if the reads/writes are linear blocks, because by the time the output uP is responding to "data ready", the input uP has finished writing the first few bytes and is almost done.QuoteThat sounds like a lot to choose between?Sorry I didn't understand what a volatile storage service means

author: C
author_rank: Super Contributor
timestamp: October 16, 2016, 10:22:10 pm
content: Just like arbitration was solved in a simple manner to allow multi-level vector interrupts in the 80's,Same is true to do bus grants with multi-level request.The only hard part is removing the need to have grant cards in empty slots, and here some simple logic on buss board cures the problem.If you do not mind having passive cards in empty connectors, ribbon cable can work for a back plain.It is just a daisy-chain that is needed.

author: bitslice
author_rank: Frequent Contributor
timestamp: October 16, 2016, 10:57:52 pm
content: Quote from: C on October 16, 2016, 07:37:49 pmIt's a state machine,why do you think timings are going to slide around?

author: C
author_rank: Super Contributor
timestamp: October 16, 2016, 11:08:33 pm
content: Strait from Z80 data sheetWhich is it, you did not read it or did not understand it?

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 17, 2016, 12:21:46 pm
content: The datasheet refers to different versions of the Z80 not one version using different clock speeds. So for one version of the Z80 I think using different clock speeds could result in quite constant timings. Although I have not measured it, I did not experience any problems with the 20MHz Z80 at very low (< 100Hz) clock speeds.I don't want to study old systems too much. I don't want my thinking stuck in some obsolete algortihm and I just love to think about this stuff myself. I am not the guys that want to build (or design) from a set example. I want to make the journey myself and learn from it as much as possible. So I would prefer to talk about this stuff at a little higher level of abstraction. I want to work out the details myself (or ask here for specifics if I'm stuck).QuoteNever mind - not important. Its just something I though up (its not really original). The idea is that all devices connected to the bus register their services with the bus controller (the bus brains). One example of such a service could be a storage service (either persistent or volatile). When a device want to 'do something', it contacts the bus controller and asks for a channel to a service. It does not know what device implements that service, it just knows it wants to, say store some data. The bus controller will route the request message to the device that actually implements the requested service and the drop out of the conversation.This is all still under devlopment, but the basic idea is not bad I think...

author: C
author_rank: Super Contributor
timestamp: October 17, 2016, 06:03:25 pm
content: Quote from: obiwanjacobi on October 17, 2016, 12:21:46 pmI look at it this way, The Z80 uses Four points on clock .1. almost high2. High drop start3. almost low4. low rise startAddress Out change is on #1 with 57nsMreq drop is on #3 with 40nsTable is 20Mhz (50ns with 5ns transition time) clock.As Ns are added to high clock time the -15ns drops to 0ns and starts increasing due to change.So if you are using the 20Mhz part -15ns@20Mhz & around 10ns@10Mhz.You have the map ram delay in address time also making the -15ns a -25ns or more.I see you having a window of time with chip selects bouncing when Z80 clock is above some frequency. If MREQ causes a latch of CS then very possible to latch to capture wrong data due to early latching of unstable data.You are then counting on no unknown changes caused by CS bounceand the address valid time before WR rise of ram chip to save you.For PSOC if triggered by MREQ or IORQ this would mean waiting longer to read address to get it to work @20 that would not be needed below some clock frequency.Quote from: obiwanjacobi on October 17, 2016, 12:21:46 pmThe higher abstraction needs some foundation in old. Not the whole of old just the parts that apply to higher abstraction.When you do look at old, you often see parts that are hidden in the new.VME is an old example of parallel multi-processor, multi-DMA vector interrupt bus.A little study of Zilog Z80's documents helps you understand the basics for vector interrupts. So VME vectors are little changed, but same idea is used for System bus access the bus req chain.Looking at OLD sometime lets you see something in a different way. Cray1 for example did new address and Mreq at same time. It used longer wire on Mreq as needed for valid address.The 8212 is a look at part of a PIO.Old can make New less of a black box. Old concepts can change when more is available. New is not always better, could just be easer. I see a lot of old coming back. A lot that you see on net is the cheap & simple, not so much of the money is no object stuff. Things built but hidden behind security.Sohttp://www.z80.info/zip/um0081.pdffigure 54"I AM Addressed" called "on card enable" in figure.Previous post I mentioned this.Can be used many ways, an IO port in middle of your ram, a movable block of of things(magnitude comparator).Most old only show a magnitude comparator replacing jumpers with a dip switch.In simple Your memory management is one level now. You can have many levels.For your many Z80s, you can think two ways. One is in DAM mode or All but one is in DMA mode. In the second, if you can remove the need to be in DMA mode at times then you can still do dma anywhere as default.When you add local memory to a Z80 you remove the need to be in DMA mode when others are not accessing it.In your thinking of DP-Ram, instead of thinking of creating DP-Ram, you could think of moving some System Bus ram to some board.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 17, 2016, 08:01:56 pm
content: Quote from: C on October 17, 2016, 06:03:25 pmInterseting. What things exactly?Quote from: C on October 17, 2016, 06:03:25 pmI think I said something similar a couple of posts back: "I could do the same [as DP RAM] using conventional RAM" (is that what you meant?). But this was when I thought of having only one buffer per exchange (session) in a DP RAM. If you allocate multiple DP RAM buffers per exchange (session) than it becomes more optimal - the writer can write all buffers available for the exchange and when finished can check if the first was read and (re)use that one to write the next block etc. (round robin).I have to get my thinking cap back on because also a FIFO memory could bring some advantages to the design I haven't thought of before. Downside of DP and FIFO RAM is that they're pretty expensive compared to conventional RAM. So I wouldn't mind designing something with (fast/cache) conventional/cheap memory.Also been reading a bit about the conveyer belt mail stop thingy. I must say it is a clever design (for the time), but I think it only works because the 4MHz Z80 was so slow! Having a 20MHz Z80 would require a bus clock that is 5 times faster (100MHz instead of 20MHz). So as the CPU speeds become higher, the feasibility of such a bus system becomes less, I feel. Also I don't like the idea of pumping empty data bins (buckets) around (48 bits wide) most of the time...

author: C
author_rank: Super Contributor
timestamp: October 17, 2016, 08:16:51 pm
content: Just a quick DP-RAMVery expensive and SMALL the advantage to buy it is normally port fight is at byte level and the hardware savings.A lot of Old computers had DP-Ram built from common memory.If you have a dual port Ram board for example, one connector to system buss and other local then the conflict normally happens at ram size of board.This is not as big a problem as some here have made it. To do dual port costs time to change ports but can be full speed between changes.The One late to the party just thinks it is slow ram for first access.Very common in cheap to build video memory as dual port. One of mine does dual port for video & char rom that is two dual ports.the big computers dual port ram cards were not uncommon.

author: bitslice
author_rank: Frequent Contributor
timestamp: October 17, 2016, 08:42:27 pm
content: Ah, you mean "Address valid to /MEMRQ Fall" is -15ns @ 20MhzIt's OK, I have no 20Mhz CPUs, I gather the ebay parts are all fake.Quote from: obiwanjacobi on October 17, 2016, 12:21:46 pmAh, neat idea.sounds a bit like something used in C++ ? Like classes define everything they do or somethingSomething vaguely related to that idea - I've got a few Context Addressable Memories, they are used in network switches to look up MAC addresses. But they are also used in database search hardware.Basically you ask the RAM if it contains a string of data/pattern, and it'll tell you yes/no instantly.I was going to use it to determine which node was working on which bit of data, or to track if an object was still current in global memory.Maybe that's of use to you, a bus controller could ask all its peripherals if they can handle, X, Y and Z, and each peripheral could do that self interrogation instantly....it's really kind of a solution looking for a (new) application, but I think parallel data searches are cool.--I also want to think up something new rather than rehash a S-100 bus, even if it's wildly overkill in terms of hardware and completely pointless.Quote from: obiwanjacobi on October 17, 2016, 08:01:56 pmIt shouldn't do, that timing was for a huge system with 256 processors, if you are using say four Z80's then the speed required to shuffle them around is only a few Mhz if that.more cheap FIFO's, it says 32K*9, but datasheet says 64Khttp://www.ebay.co.uk/itm/281866493589

author: C
author_rank: Super Contributor
timestamp: October 17, 2016, 09:58:48 pm
content: Quote from: obiwanjacobi on October 17, 2016, 08:01:56 pmWorked on many a radio receiver.One was three 6' racks of electronics. Short answer, Two RF inputs at same frequency with remote tunning to HZ. Think manuals for the thing was 3 file drawers, so lot missing from description.A radio receiver with with two PDP-11s & Z80 plus more.Wire storing data because nothing else was fast enough back then.An audio amplifier driving a ac motor. Not a big deal, until you add the amp was tubes and 1500w. nice 1500v supply in it.Lost count of the types mini computers I worked on.Contractor said you can use our 200k$ mini computer when we are not, You break it- you fix it.Delayed me getting my first micro for a bit.Quote from: obiwanjacobi on October 17, 2016, 08:01:56 pmI said it before, let me say it again!! Working between two cpu's is like working between two sets of interrupt code. When you add that the Z80 has NO read-modify-write as one memory cycle then you can not have the two computers writing to the same byte. You have to drop back to One computer writes & controls the byte, all others just read it.circular buffers as used now between main program and interrupt code work fine. You have a buffer for OUT and a second for IN.You have probably worked with circular buffers when dealing with async serial.In stead if thinking characters, think of array indexes. You have 8 bits, a lot of locations or type & location. The array location could be a pointer to anything.When you have code to work both directions of a circular buffer, not hard to do a large number. Think add a processor add two buffers and not a bunch of code at this level. Two circular buffers is like two FIFOs with software size adjustment.Think of what happens when one cpu can talk to just three places. You have a tree of cpu with the total count one cpu less then the next level.Now change it to one UP and many on a bus. Grows faster with less levels. The actual delay to pass through many levels might not be all that large, but if it is you all ready have parts to cure problem, an added link.Think of that 256 Z80 ring.You have a back plane, join the back planes in a circle with system bus links. It's slow to talk to 128 away so add an inter circle. you could have an inter circle of nothing but system bus links plugged.Something that is small that can scale is a powerful tool.If you think it through an 8-bit pipe has a huge overhead inside a computer if used wrong. If you send binary programs through that pipe then every bit combination is already in use. To get a second set of data through you have to replace a byte to have a key value to change on. To replace a byte you need a byte. You are saying when you see this byte it is ___ , but you then need a second byte value to put the first back.a second byte that says when you see this byte replace it with ___.The two bytes can not exist in the stream with out problems so the byte following replace byte is array location to get byte value to replace.You are now scanning all input data to replace bytes as needed and processing output stream to put it back.Think of this page you are looking at for an example, or simple XML.Do you want this overhead between two CPUs just to talk as needed?For a Z80, things just went to a craw due to all the processing needed.Change to a 9-bit pipe and you have no need to check program binary as it can not have a 9th-bit. Packet systems that use 8-bits have a marker that can not happen in the data to identify start of packet.If you have a packet link you can send other packet data over that link. Packets in packets or packets that have sub packets like Ethernet does with IPv4 & IPv6SCSI and others use a 9th data wire or data bit to separate what is data from something else.When you have a solid packet start point then length fields will workSorry Long again

author: C
author_rank: Super Contributor
timestamp: October 17, 2016, 10:50:26 pm
content: Quote from: bitslice on October 17, 2016, 08:42:27 pmThat system would only be good for little input with more computing.Would spend a lot of time just waiting for other types of compute loads for a net loss of processing. The only good point I see is a known fixed delay between CPU's, Not all the same but fixed.Think of how well an idea works for an an old parallel printer.For a good Z80 interface, writing to a IO port caused that data to pass all the way to the printer. The printer controlled the speed and sent back status information. Reading one IO port got you the status of printer and if ready for next character. Adding interrupt drive was easy. So you could have a circular buffer and have option for file based print spooling. If you built the software correctly then it was easy to add print filters, things like changing tab's to spaces or linefeeds to CR-LF or even one set of print codes to a different set.In a real computer you have many different things, all most the same but just a little bit different.Little things can make a big difference to user program. A fixed size FILE BUFFER for user program takes some load off the os but also means that user program is now having to do extra work when data Size is a different size. If the OS can handle a buffer of 1 to 256 in size it makes a lot of programs easer to write and could be faster as well. But you just know that some user program is going to want a size larger then what ever the limit in the os.For a raw data stream the double buffer is a cure. A better cure would be to tell the os that a circular buffer of this size exists and ask for more at a sub set of that size. Sounds like video from YouTube to local buffer.Putting a hardware FIFO in does cure some problems but you could easily end up with circular buffer connected to fifo connected to circular buffer for a net loss. And still run out of hardware Fifos.EDIT: sorry left out file buffer above

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 18, 2016, 08:19:28 am
content: Quote from: bitslice on October 17, 2016, 08:42:27 pmAh, right - of course. I think that 16 is a nice target: 8 CPUs and 8 IO devices; 8 MHz - 10MHz should work fine.About how to interface the software, using ringbuffers or what not - I am not at that stage yet. I am still trying to paint a hardware picture and how the data is efficiently exhanged between devices. You (C) mentioning double buffering did spark some inspiration with me with having a minimal of two data buffers in a DP RAM dedicated to an exchange. That could work well I think.Using DP RAMwillmake the exchange asynchronous so writer and reader can work at their own speed. Still, they have to sync up to communicate block-ready handshakes... Not thinking about block retries yet, the happy flow first.I was also thinking in making the bus 16-data-bits wide (brain-breeze). It would mean moving twice as much data with the same effort, however the interfacing with 8 bit CPUs will introduce additional overhead. The question is, is it worth it (just thinking out loud).

author: C
author_rank: Super Contributor
timestamp: October 18, 2016, 08:22:44 am
content: Quote from: obiwanjacobi on October 17, 2016, 08:01:56 pmStill think I see some miss understanding of DP ram from standard ram.Will make some dumb steps here to help with later parts.Lets make a simple 256 byte ramcardto start with using good design practice, for a 64K Z80 system.This means buffers on all signals coming on board in this case.You have three 244's connected to address lines and control lines.You have a 245 for data lines.I hate jumpers and at 256 bytes could need a pile of boards so use a magnitude comparator with a dip switch for upper address decode."I Am Addressed", (IMA) = output of magnitude comparator & buffered MReq. This assumes address valid before Mreq..A = IMA245 data bufferDirection IN & not tri-state = A & buffered Wr.Direction Out & not tri-state = A & buffered Rd.Lower address 244Not tristate = AA_Ram = ARam OE = A_Ram & buffered RdRam WR = A_Ram & buffered RdRam CE = A_RamOk at this point if I did not mess up ram & buffers work, A simple ram card, some tri-states not needed.Dual Port ram chip has two ports so a copy of above for Side B with some tri-states removed. Spend $$ for DP RamThe gold is to again use standard chip.Now if you look at above, when A is not active all side A to ram is Tristate.B is a copy of A so Both sides A & B can be connected to standard ram chip as long as only one is active at a time.In stead of trying to add wait states, lets do the reverse and remove wait states.Z80_wait active = IMA going active for that side.In time sequence is1. ram idle2. IMA active starts cycle3. Arbiter make side active, A or B4. tri-state Switch has finished changing.Proper Address & data to ram chip.WR may need to be delayed to make ram happy. (Data-in to WR = 0ns with no delay).MREQ delayed to get back to address before Mreq.6. Ram cycle starts.7. Active Wait to Inactive at proper time for ram8. IMA to Inactive ends cycle for ramA & B to inactive9. back to 1A delay can be just inverter to inverter to get 2x inverter delay. if not enough delay, add in pairs in signal line.So we have these states from input buses(Not IMA_A & Not IMA_B,IMA_A,IMA_B,IMA_A & IMA_B).This is a problem area as race conditions can happen.Only one side Requesting do cycle above.If second side is Requesting before end of cycle, you want a side change for round robin.Note that with round robin, both sides see a min memory time of a cycle above and a max of two cycles.If both Z80's tried to do a word read, it would cycle A,B,A,Bor B,A,B,A.Notice that if bus A is active, B has to wait and if B active A waits, A 256 byte address area.Arbiterhttps://en.wikipedia.org/wiki/Arbiter_(electronics)You can use a clock or digital delay line to make it easer.Now if you are still worried about a slow down, You can make it less if you use two of above boards & swap A0 & A8 on both boards. You now have odd & even boards.ABX board oneXAB board twoNote: Any size Ram works, Two or more Dual ports made from standard memory works a little better then one of same size. One is fine.NOTE: you actually have real memory inside so anything can be inside connected like the ram(IO ports).Each CPU Board connected to system bus has a version of A !!! with small changes

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 18, 2016, 08:55:14 am
content: Quote from: C on October 18, 2016, 08:22:44 amWhat do you think I don't understand?What you describe is more a mux than a dual ported RAM. It's a way to share one RAM between two users.If you would make the physical RAM chip size relatively small, say 4 or 8 k, then you could switch them indiviually and simulate a DP RAM for data block (of 4 or 8k) exchange.

author: C
author_rank: Super Contributor
timestamp: October 18, 2016, 09:24:55 am
content: Quote from: obiwanjacobi on October 18, 2016, 08:55:14 amThat is what a dual port ram really is just mux in and out and some timing. so simple very cheap computers use it.Same idea is the system busThe A part is on the CPU card.The ram is the system bus.and the timing is the bus grant ArbiterWhen a system bus CPU uses 245's in place of the 244's in A very little logic is needed for system bus access to cause what looks like a on board DMA to local CPU.A dual port ram card can use any size memory. I have seen 4 Mb boards and 2k boards. You gain a little using more boards.Two areas of dual port gains you more just like system bus on one port and local bus on other.It is also easy to run the Ram at faster then CPU speed, Above 2X very little conflicts. Very little change to do this, just two latches to capture Data from ram.Small changes applied at the start can make a huge difference.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 18, 2016, 09:49:53 am
content: Quote from: C on October 18, 2016, 09:24:55 amThe arbitration in a DP RAM is done at byte level not at bus level. So the granularity of the conflict is much lower allowing for much more parallelism.Quote from: C on October 18, 2016, 09:24:55 amWhat I described is a way to build your own poor-man's DP RAM where the granularity of the contention is at block size. So its better than bus-level contention but worse that true DP RAM byte-level contention - but also much cheaper. My reasoning is, If you do not really need the byte-level parallelism of a true DP RAM, why spent the money for it.It has nothing to do the total amount of local or system RAM. This (poor-man's) DP RAM is only meant to help make the data exchange between devices faster. I think it's useless to make the DP RAM too big, you aint gonna need it IMO.

author: C
author_rank: Super Contributor
timestamp: October 18, 2016, 09:58:34 am
content: Goofed read previous post again.Looks like you got DPMost of the TurboDos slaves had about 2k of DP showing on system bus.So that size can work. Note that you can use huge ram while showing just a small amount to system bus. With volume a 512k could be cheaper then buying some small rams.If you use magnitude comparator's with a port controlling one side for decoder .IO or Memory can move, be hidden or overlay existing things at magnitude comparator address decode.A window just needs two >= & <. If one bit is fixed instead of address line, you get show or hide.When shown, that magnitude comparator output can be used to inhibit what is normally there.A port that shows some place in one of your existing ram's bytes, just needs an inhibit all input to existing decoder for existing ram chips.Master was like your PSOCBetter slaves had one or more IO ports showing on system buss.Trying to remember hereSome could interrupt master and master could interrupt slave.Some had a byte for interrupt vector to create.Your hold reset on power up was in IO port to master and could reset slave at later date.Think really really huge. If you do not want to do huge now put in pins with jumpers, 0 ohm resistors or just pins or leave parts off.Might also think of 16-bit wide circular buffer still in 8-bit ram. Not sure which is better input two bytes each time or two 8-bit circular buffers used as a 16-bit wide. Quick thought is 2x8 when dealing with 2 cpu's.One thing, Input side should put all data in place before putting last byte saying it is there. Take out all then say got it.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 18, 2016, 03:52:35 pm
content: How the bus RAM used to exchange data is mapped into a device is the problem of that device. For my CPU board that has a MMU and 4x 64k on board, the method is set by the design of that MMU. Other devices (that I still have to design) may use a totally different way depending on what is fastest / easiest for that specific device.That is also the reason why I do not want to design the interaction in such a way that it depends on a specific feature or trait of the Z80. I should be able to hook on an Arduino and get it to play.

author: C
author_rank: Super Contributor
timestamp: October 18, 2016, 03:56:56 pm
content: Quote from: obiwanjacobi on October 18, 2016, 08:19:28 amA good design does not have limits, A back plane has limits due to driver/receiver and distance and can be connected with repeaters.QuoteNot realy, with circular buffers you can check the free space & how much in in circular buffer. CPU using data is just asking more to remove polling and just needs status back, no sync needed.double buffer is the min, circular buffer can be 2x to ?x of blocks.circular buffer in, circular buffer out as message lets both ends work. A circular buffer is a Great way to keep track of a bunch of linear buffers for free and used.Just think always the oldest empty buffer is used for new data.Block data is just a circular buffer entry pointing to a structure that contains the facts an has the block in it or a pointer to a block.QuoteWith a good design any 8-bit cpu can function on bus as they all use standard memory.You can have some 8-bit bus and some 16 bit bus this can make it easer for the 8 to 16 logic.The catch is you do not know when or what size with a 8-bit cpu, everything is byte even if it's to bytes.A 68k is really just using TWO 8 bit buses With a common address bus. A0 gets hidden in one byte or two byte logic. The two WR & RD's are used separate.The problem area is using specific family IO chips with different family CPU.As you increase Z80 speed, You start losing what Z80 io chips will function. Can not think of one that works at 10 or 20 Mhz.There is some chips almost same from Zilog that will work even on a 68k at the higher speeds. some are better then the Z80 IO. Might be a large address DMA chip in these.68k dma could give you 24-bit address range, your mapper probably limits each transfer to mapper size.But A mapper could be used for system bus <-> local bus and problem is hidden again.

author: NivagSwerdna
author_rank: Super Contributor
timestamp: October 18, 2016, 04:39:14 pm
content: Try reading this...http://www.trevormarshall.com/old_papers/Floppy_Disk_Transfer_Techniques.pdfOnce upon a time I wrote some Z80 code (part of the disk reading software in MSX-DOS for the ASCII corporation), in those days the word "fast" had a different meaning, paging in blocks of 64k was hard enoughGo simple.

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: October 18, 2016, 07:30:51 pm
content: Quote from: NivagSwerdna on October 18, 2016, 04:39:14 pmWhen you have a slow CPU and memory bus, you don't want to waste cycles unnecessarily. In the old days we did it because the cost of efficient hardware was prohibitive. Now we have cheap CPLDs and MCUs, so that is no longer an excuse.The data transfer method proposed in that paper has two instructions in the loop (INI + JR NZ) which takes 7 memory cycles per byte. A good DMA controller could do it in 1. A good DOS would transfer the data directly from the I/O port to the final destination and not go through any buffer memory.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 19, 2016, 06:09:26 am
content: Quote from: C on October 18, 2016, 03:56:56 pmI totally donotagree. I experience design as making choices in the direction that you think is important. That means that you choose to live with a certain limitation, in order to make something else (more) possible.Quote from: C on October 18, 2016, 03:56:56 pmBut that is not the point. The point of this bus is to elevate the IO (and cross-device communication in general) above that physical implementation level (of support/peripheral ICs) in order to off-load having to deal with the IO-specific details from the CPU (that requests the exchange). Thatdoesmean the bus must be as fast and efficient as possible, because higher abstraction levels usually brings some overhead with them.Quote from: Bruce Abbott on October 18, 2016, 07:30:51 pmAnd DMA could even go faster than the CPU clock if the (async) RAM speed would allow it. For instance I have some 10ns RAM on my board which can be blasted way faster than the Z80 is doing.Problem is how does the bus know how fast each device can handle? So perhaps the clock for the DMA data transfer should be provided by the device?

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: October 19, 2016, 07:57:52 am
content: Quote from: obiwanjacobi on October 19, 2016, 06:09:26 amOr wait states inserted by the device or allocated according to I/O or slot address, or programmed into the DMA controller before each transfer, or each device has its own DMA controller. The bus itself doesn't need to know how fast the device is, it just responds to whatever is controlling it.But before designing a complicated bus system that can handle any possibility, ask yourself what youreallywant from it. If it's just to have a complex system that can handle any possibility (with no particular purpose) then it might be best to stop now before you waste too much time and money on it. OTOH if youdohave a particular application in mind, then design the hardware to meet that purpose, and don't worry about other possibilities that will just make it more complex and less likely to work properly.QuoteIndeed.Everythinghas limitations, and trying to design something that has none is bound to fail. Something may 'possible' but not practically realizable within the restraints imposed. For example it might be 'possible' to have 8 CPUs and 8 I/O devices each on its own card with a bus that can cycle at 10ns, but in practice would be very hard to achieve. - and if you don't have a goodreasonto run 8 CPUs in parallel then it's just a waste of resources, even if you could get it to work.

author: C
author_rank: Super Contributor
timestamp: October 19, 2016, 07:25:08 pm
content: Quote from: Bruce Abbott on October 18, 2016, 07:30:51 pmWoW, guess you have not heard of storage devices that use different size blocks. Then Every user program is to adapt to 128 byte to 4K blocks. Then you have 64k bytes a few years ago and all programs are a fail again.Some Smart people have said learn from history.You wanted higher abstraction so here.You need no OS, you could do it all in the program.OR you could have a huge OS and do it all in OS and have very little program.Or have something between the two.For some parts does the User program set the limit or the os.First versions of CP/M put file buffer in user space with a 128 byte size for some good reasons at the time.So programs stayed compatible while the OS in later versions handled the blocking needed in OS for 1(128),2(256),4(512),8(1024) blocks per physical sector. Some hard drives used 4096.A simple change to 256 will break all old user programs. Do you make the OS handle both? That's is a new set of OS calls.If someone different made A New OS they just demanded new programs or rewrite of all old.The small size file buffer has it's good points and it's bad.A disk based sorting program could use more files to make the sorting faster. A text editor can easily use the small size.Fast streaming is easy as very little costs get you many blocks head of output data. Here the important question is can OS do work while user program works.The user program can do what the OS can not, if the OS does not prevent it.How many reached the level of use and time length of CP/M.The PC did not win on being a better computer because it was better then others. A good part was easy translation of 8080 & Z80 programs.There was better CPU's. When PC was introduced I was using over a 1Mb on a single user system. The computer center asked my boss to start programs later in the day and tell them ahead of time so that they could do some changes to handle the load.Think it through with a user program using two files working with 64k memory. At 4096 the OS just about had to do read-modify-write of a sector. And a lot of programs used more then two files.Might be a good idea to know why something is/was the way it is/was and what limits they were working with before calling something trash.Different limits make new options possible.Quote from: obiwanjacobi on October 19, 2016, 06:09:26 amOK said that bad. Trying to say how is 8 now is such a limit, why 8? What now is imposable?No hardware limit of 8 that I see. An not far enough along for a software one.When you set the limits to low then they come back to bite you. Often sooner then you expect. IBM picked 4 which to some was a problem very quickly. At the time 256 or greater was just as easy, An artificial limit set to low.Way back in time a S100 back plane, The IC's started putting a limit on slots at 16 slots, was not to bad to deal with, With care you could get to 20 slots. With the newer high speed cmos IC's more then this is a could be possible. Simple circuit could connect back planes. You have hardware limits & you have software limits, have you reached a point where you know 8 is hard and fast or is it a what you see now.Reasons and how, they do make a difference.Quote from: obiwanjacobi on October 19, 2016, 06:09:26 amTrying to say that for some of your smart io, two things could be happening.The io cpu could be running at a different cpu speed to be able to use slower chips to make smart io better. By having many smart io you can have one running at clock speed x to use io chips that max at speed x. A different board could be speed Y. When looking for an SIO you have many choices. A Z80 sio adds vector interrupt that others may not, but often missed is the other zilog sio chip that can work with more CPU's . Need an SIO for a Z80 look at other Zilog chips, I think other here is better than Z80 SIO and with higher speed possible.Short version Z80 Sio or Z85C30Smart IO is not new, way back it was SCSI,GPIB, networks & point to point links that could scale HDLC or IBM's SDLC.My first network use in 1974 went to points around the world back then. WAN is very old idea, just like connected wan's are. Should I also mention it was encrypted at link level.To me, This is all very old stuff.What your mapper does is not new, just more limited back then due to parts speed and cost.Quote from: NivagSwerdna on October 18, 2016, 04:39:14 pmNice paper, Would have saved me some time putting all that together. I wonder how many got caught in the 6mhz CPU with no 6Mhz DMA yet.Just info you might like, TRSDOS moved directory to a middle track and used sector mark change as backup to find the track.Smart IO is great when done right and can scale. Wrong it's a pain and limiter.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 20, 2016, 05:56:49 am
content: QuoteBut what 'right' is, is different to everyone (personal preference, style and beliefs) and different for every situation. Right is specific to the environment and the function the hardware is required to fulfill.I am probably going to use MCUs for serial interfacing. Cheaper, more flexible, more choice etc. But I have not come to that level of design yet.Strikes me that a lot of EEs design at the hardware component level from the start. I like to design in 'logical function blocks' and then flesh those out. I sometimes consider how something abstract would work on hardware, but nothing in great detail. I do also sometimes consider the implications for the software - how would it be to write code to drive this hardware. When I have the bulk thought out, I then start to fill in the hardware and may have to reshuffle e bit here and there but at least the 'vision' is clear and set (no feature creap or function drift). If I deviate from the plan, I know exactly why. For some blocks I know up front I have to use a specific part (like the DP/FIFO RAM).For instance for the bus, I have a part of the software protocol on paper. This I can use as an example to check if the hardware is up to the job or that I need to change something to a more optimal solution (at hardware level).So at this stage I am looking for paradigms, algortihms, architecture, best practices, mechanisms and patterns of efficient data exchange.I am sure you'll find a lot wrong with this approach but this is the only way I can keep a lot of detailed information structured in my head.

author: C
author_rank: Super Contributor
timestamp: October 20, 2016, 09:18:41 am
content: Think of a floppy disk controller.Smart to some means put file system in floppy controller. If you do that it's a fail in my book. My floppy controller worked with many file systems. You make smart a caching floppy controller, it could be a win. But there is a third choice. Go back to simple controller and put ON on controller. Have OS on z1 talk to os on z2 that has controller. z2 can cache sectors and might do part of os also. Just really changed one thing os to network enabled os.z3 could do printer, z4 console all with just that one change.Think of iscsi, scsi over ethernet. original scsi used a 9-bit to separate control & status from the raw data. To change this to ethernet you could add a length field stating control/status is this long & a length field for raw data. Very little cpu required. This page you are looking at has display data and text, it takes a lot of cpu power to get from 8-bit stream back to seperate display data & text. Every byte in the stream has to be compared to many byte values to separate the stream.So think you are a C person. Think of communicating in C structures, First data in structure is Length of structure. second item is ID# for structure. Even if a middle man in path has no idea of structure contents, middle man has a length and can pass structure on. This is assuming a serial path, parallel you use structure directly.The same idea lets you split a user program to use more CPU's in series or parallel.You need just a few parts. system bus memory or system bus with DP memory local connection & or local memory acting like system bus memory.Spend $ to buy DP or build your own. When building if you can do two ports, more ports is not a big step. A building block with power.System bus, Think of what it is at small level.One of your Z80 boards using memory of second Z80 board while both running programs. A DMA into send board with DMA controller being the Z80 on first board. Connect the two boards with 245's (hint use two for control). Figure out the logic needed, it's just two pins on 245''sThink you said you have 4 64K on board, to start simple 1/2 of Z80 address map is other board. When you have basics, address and range are easy change.I work from the bottom & the top as the very bottom makes top possible. Make a list of all the bottom ways that you can connect, then look at top.URL line is not bad for high. A system that has GPIB which has addresses then gpib address becomes a sub address.For a network of systems, a system that lets you work up and back down makes connections easy. For unix/linux the dot & dot dot relative file address for example.Much of a protocol will kill a z80. Need to move blocks of raw data with little compute power. Think DJMZNote I have said 9-bit data many times. this can be very easy on send side.An output port with two addresses, A0 becomes the 9th bit. For Input you put 9th bit in a status register where it can easily be read, A simple shift to a flag you can branch on.Quote from: obiwanjacobi on October 20, 2016, 05:56:49 amThat is why I pointed you to turbodos, It works, is fast on Z80. A huge amount of thought went into it that you only see the top in the implementers guide. Can not think of a second example you can get your hands on. The big problem was some of the fields in the structures are too small for today. When you use CP/M limits, turbodos's same field is larger. A CP/M system with some of the limits would have no user program space left, while same system in turbodos network had more user space for program. You could run a user program and have it customize the system for next program to run working around CP/M limits. You could chain CPU's to do a task.Look at each structure and ask what is it used for and when. You have the OS on top using the structures & user program on top of that.If turbodos was not maintaining CP/M comparability, even more could be done. They did what they could by adding turbodos calls in addition to cpm calls&mp/m calls. All before a PC existed.I gave you a list of building blocks needed for super system based on system bus. Remember that back then memory and disk drives were money so programs were bulit to work in limits and get job done. And not was time doing so.Today I think it' throw more cpu,memory & disk space at a problem.Edit: Thought I should add, Today the equal of TurboDos is all broken apart into separate parts. You gain some things but lose small, simple & effective in most cases.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 23, 2016, 06:37:32 am
content: QuoteNope. Although I started in C, I have been programming for about 25 years in OO (C++/C#) and it is really hard for me to design a decent API in C. I think its a crummy language that only works well for small projects. I am sure that if you have a lot of experience, you've found ways to make it manageable, but it doesn't do it for me. If available I always go for the C++ compiler (for AVR stuff etc). Unfortunately thez88dkI plan to use, only features a C compiler and an old one at that.The pattern that I now use is that I still think in objects and implement them in C. And with some strict naming conventions and the use of structures it's sort of decent. So say you have a Car object, then you'd have a Car.h and a Car.c file. All methods start with Car_xxxx and then the method name. That makes it clear that these methods belong to the Car 'class'. For instance data members you'd have a Car struct that contains the vars. Each Car_xxx instance method would take that struct as first parameter (this ptr). I try to stay away from virtual functions, they're messy in C and in the small confines of an MCU or a Z80 you would want to avoid them anyway. I also try to have no globals, statics in a 'class' are ok - only to be accessed using the class-specific functions.

author: C
author_rank: Super Contributor
timestamp: October 23, 2016, 10:05:44 am
content: Not a real fan of C or C++ as I got to use a pascal that had modules and extensions for system programming back in 80's. Using it you had a case of being able to write a little more source and gain a ton of error checking by the compiler. Was not perfect but sure was nice.You might want to do some web searches for some pascal like languages that compile to Pcode. You could get Pcode running quickly and then change compiler to also output z80 code. And the reasion I am mentioning this is that additions to pascal like compilers is not that hard to do. There is a full operating system written in oberon (pascal like) and a compiler that outputs to C. A little work up front but could be great in long run. The os could be to much for a Z, but you would have full source to make changes to. You can try os in virtualbox.Your thinking of working top down, but some things are best thought in reverse. You might be thinking of not using DMA for example. I would think in reverse if not a major cost. This is one Z80 to memory area of another. One Z80 runs and all others are in bus grant state. Not good for speed, but then build a layer above this that makes it just one om bus grant state. Slap DP ram on top to make this happen even less. You now have all kinds of software choices.Remember you can have I/O in memory space. One advantage is 16-bit load/store to I/O in one instruction.If you looked at figure in previous post My "I am addressed" is in that logic as "OnBoard". Do not know if this is possible on your boards now, but a address decoder output of OnBoard will make connection to other boards easer. An if you can, increase decoder input to have one more address line. With a resistor pull up, your board functions the same as now. When pulled low nothing on board responds. So simple logic is if Z80 is accessing something off board at any address, pulling that one line low prevents onboard stuff from responding. Z80 is working 0-1M always, Onboard is 1m-2m with resistor making this default.With this foundation Your high level thinking could be any address to any address and with another small change, one z80 being able to send a vector interrupt to any other z80( all 127 vectors to each).User CP/M program is a make request & wait. Parts of TurboDos is many requests with response as fast as possible but only in time sequence for a user program. Next step is user program being able to make many requests with response being able to be linked back with request.You need an event based language to make this easy. Interrupts in C could be a way to function between Z's.For high level block devices think of a remote floppy disk controller to make you think of time delays.

author: ebclr
author_rank: Super Contributor
timestamp: October 27, 2016, 08:19:54 pm
content: DMA & Ide together is a point to starthttp://www.retroleum.co.uk/electronics-articles/an-8-bit-ide-interface/