PCI Gigabyte I-RAM without the PCI part :) - Page 1
Forum section: electronics / projects

author: gnif
author_rank: Unknown
timestamp: January 15, 2015, 05:34:51 am
content: Hi All,I have always wanted a hardware ram disk to play with, and when the Gigabyte I-RAM came out I was amazed at the performance that was possible with this device. These days its performance is not that great in comparison with SSDs. The reason for this hack/mod was because I recently had to rebuild the RAID array in my home server after a drive failed, since this was an opportunity to update to ZFS, I took it. After learning about ZFS and how you can provide it with even a small USB (slow) cache disk for massive disk performance improvements, I decided I would see if I could obtain a 2nd hand I-RAM.After a bit of searching I found one supplier in the US that had a brand new I-RAM listed as unsold stock, I snapped this up for $40USD. I already had 4x 1GB DDR modules from an old upgrade (Yes, I hoard), so I did not have to purchase those. Next problem was that my server has no more PCI slots remaining on the board, they are all in use. After a close look at the I-RAM (about 2 seconds), it was very obvious that the device is only powered from the PCI slot, there are no signal lines connected except for RESET.I dug out an old faulty motherboard (yes, again, I hoard stuff), and did my best to de-solder a PCI slot. Now I realise I could have just soldered onto the I-RAM, but I would like to keep the card in good condition, it is an item of interest. I got the PCI slot out, but lost 1/2 the pins in the process, which was no big deal since we only needed to provide the +12v, +5V, +3.3V and ground connections, and there were enough pins remaining for this after rearranging them.It should be noted that this thing has a battery on it, and charges it, also DDR can be pretty power hungry, so to be sure I could supply enough current every ground and voltage pin must be connected (later I confirmed this theory). In total about 20 pins had to be wired up, and a small DC-DC switch mode regulator to provide the +3.3V rail.After plugging in a test card (random PCI sound card) to ensure there were no faults (there were, had +5v shorted to GND), I installed the I-RAM into my adaptor. LEDs came on... great! After noting that it draws 1.2A, I was glad I opted to connect every power pin. Next I wired up the Molex connector and attached it to the PC, lights came on, and an extra one came on when the SATA cable was connected, but no joy, the PC would not see the device. I did a quick double check of things and I decided to pull up the RESET pin to +3.3V. Success, the device was now accessible.Code:[618884.466560] ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300)[618884.467076] ata6.00: configured for UDMA/133[618884.467084] ata6: EH complete[618884.467186] scsi 5:0:0:0: Direct-Access ATA GIGABYTE i-RAM 8 PQ: 0 ANSI: 5[618884.467398] sd 5:0:0:0: Attached scsi generic sg3 type 0[618884.467409] sd 5:0:0:0:  8386559 512-byte logical blocks: (4.29 GB/3.99 GiB)[618884.467561] sd 5:0:0:0:  Write Protect is off[618884.467565] sd 5:0:0:0:  Mode Sense: 00 3a 00 00[618884.467647] sd 5:0:0:0:  Write cache: disabled, read cache: enabled, doesn't support DPO or FUA[618884.469121] sdd: unknown partition table[618884.469430] sd 5:0:0:0:  Attached SCSI diskNow as I said earlier, the I-RAM is nothing to brag about these days, benchmarks show a consistent 120MB/s and 0.1ms access time. Any decent modern SSD can outperform this device, but the down side of the SSD would be the write life as the I-RAM should electrically fail before it hits any write limits, making it the perfect RAID-Z cache device on a budget.Since these photos were taken I have covered the connector pins in hotglue to prevent any shorts or movement as this connector has seen better days after my treatment of it while de-soldering (levering out with a crowbar?) it. I have also clearly marked the front end of the connector as the card can be plugged in backwards, don't want to blow it up should I need to change things later.Code:xen:~# zpool statuspool: datapoolstate: ONLINEscan: scrub repaired 0 in 5h45m with 0 errors on Sun Jan 11 07:45:19 2015config:NAME STATE READ WRITE CKSUMdatapool ONLINE 0 0 0raidz1-0 ONLINE 0 0 0ata-WDC_WD40EFRX-68WT0N0_WD-XXXXXXXXXXXX-part2 ONLINE 0 0 0ata-WDC_WD40EFRX-68WT0N0_WD-XXXXXXXXXXXX-part2 ONLINE 0 0 0ata-WDC_WD40EFRX-68WT0N0_WD-XXXXXXXXXXXX-part2 ONLINE 0 0 0cacheata-GIGABYTE_i-RAM_XXXXXXXXXXXXXXXXXXXX ONLINE 0 0 0

author: Fsck
author_rank: Super Contributor
timestamp: January 15, 2015, 06:43:51 am
content: if your ARC throughput is 120MB/s, that doesn't seem like it'd accelerate transfers much unless you're doing small size, high iops predominantly.

author: gnif
author_rank: Unknown
timestamp: January 15, 2015, 06:49:34 am
content: Quote from: Fsck on January 15, 2015, 06:43:51 amYou are correct, this pool is for a development server and local caching proxy, lots of small files. Honestly though I do not need this, it is just for the cool factor

author: Scrts
author_rank: Frequent Contributor
timestamp: January 15, 2015, 09:36:12 am
content: Thumbs up, nice work!I wonder did anyone ever made a faster device with faster memory? Maybe something for servers? E.g. DDR2 memory stuff before SSDs came in.

author: gnif
author_rank: Unknown
timestamp: January 15, 2015, 11:28:33 am
content: Quote from: Scrts on January 15, 2015, 09:36:12 amThanks. Yes there are other offerings that use DDR2 and DDR3, but they are extremely expensive, usually only used in high end database servers that require insane IOPS. These cards interface directly over the PCIe bus, no SATA interface on them, I believe Intel recently started offering cards that give throughput in excess of 1GB/s.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: January 15, 2015, 01:03:21 pm
content: Quote from: Scrts on January 15, 2015, 09:36:12 amI don't really see the point. Just throw more RAM in the server and mount some of it as a ramdisk in the OS. 10+ GB/s for just the cost and complexity of more/bigger RAM modules. I do this all the time on my servers.Back "in the day" when motherboards could only support a few GB of memory (or less) max, this device made sense, but that hasn't been an issue for a long time.

author: TiN
author_rank: Super Contributor
timestamp: January 15, 2015, 02:56:23 pm
content: Been there, did that.More straight forward tho, just SATA power plug to cut pieces of MB.There is another option, with DDR2 and dual SATA3 ports - ACARD ANS-9010, based on Xilinx Spartan FPGA.

author: gnif
author_rank: Unknown
timestamp: January 15, 2015, 08:55:49 pm
content: Quote from: suicidaleggroll on January 15, 2015, 01:03:21 pmThese devices are battery backed so can be used for persistent data. Also you may be (like me) running in older hardware that is already running with the maximum ram capacity. And finally, software such as ZFS already use as much RAM as it can to improve performance, this just gives it some more to work with.

author: mattaw
author_rank: Contributor
timestamp: January 19, 2015, 02:41:07 am
content: It is great to see another person do this mod - as well and see all our different approaches (TiN's I saw on HardForum? but somewhere first). I documented mine here and also the need to keep some 2N7000's around in case of mistakes - had to replace the same one twice so far due to floating the reset until I got that figured out. When I read that you hadn't pulled it up I was worried for a while that you would burn it too.Here is the link to my solution, similar to yours but I used a linear from TI hotplate soldered onto a larger scrap of PCB for heatsinking and hardwired to the card itself mounted to the case lid with standoffs.http://mattaw.blogspot.com/2014/06/modding-gigabyte-i-ram-gc-ramdisk-to.htmlStill a fantastic amount of IOPS for not much money. Would love to get my sticky fingers on a A-Data box. I should add that there are some marvell PCI-e RAM/NAND Flash cards on eBay for about 30-40 USD but driver support is very lacking AFAIK. They were designed to plug into Dell Equalogic storage systems. Might try picking one up anyway for my new system.

author: Boost
author_rank: Newbie
timestamp: January 19, 2015, 07:08:06 am
content: I'm qurious. What OS are you running ZFS on?

author: gnif
author_rank: Unknown
timestamp: January 19, 2015, 07:23:13 am
content: Quote from: Boost on January 19, 2015, 07:08:06 amThis is on Debian Linux usingZFS On Linux

author: mattaw
author_rank: Contributor
timestamp: January 20, 2015, 04:31:37 am
content: Ubuntu with zfsonlinux. However I am looking at linux mint for an OS without all that Canonical "Magic" I am so unfond of. Especially the integration with amazon and other stuff which sends all your search results to commercial companies. I understand they want to make money but I will be making other choices of distribution in future.M

author: chickenHeadKnob
author_rank: Super Contributor
timestamp: January 20, 2015, 05:42:52 am
content: Quote from: mattaw on January 20, 2015, 04:31:37 amMint also monetizes their users.

author: gnif
author_rank: Unknown
timestamp: January 20, 2015, 02:05:10 pm
content: Quote from: mattaw on January 20, 2015, 04:31:37 amJust use Debian, no Canonical Magic, still pure APT, and widely supported. Ubuntu IMHO is a load of bloat ware.

author: JJalling
author_rank: Unknown
timestamp: January 20, 2015, 02:57:51 pm
content: Quote from: gnif on January 20, 2015, 02:05:10 pmMy server is currently running with 4x2TB disks in RAID-5 and ext-3. I plan on changing the disks to 4*4TB instead.Would you recommend that I switch over to RAID-Z instead of just building a new RAID-5? I have read that ZFS requires a lot of memory. I currently have 16GB installed - is that enough?BR Jonas

author: macboy
author_rank: Super Contributor
timestamp: January 20, 2015, 07:19:15 pm
content: Quote from: JJalling on January 20, 2015, 02:57:51 pmWhat is your server doing? Storing and serving media files for you and your family, or for a neighborhood of ten thousand people? If the former, then yes you have enough memory. If the latter, then no you don't.I run ZFS on OpenSolaris with 2 GB RAM. I have 5 x 1.5T disks + separate ZFS root disk, no cache device, and it performs extremely well as a home storage server. ZFS is amazing. You will love it compared to RAID-5. When one of my disks started to go and sectors started becoming unreadable, ZFS detected which sectors on which disk were bad and repaired the affected files automatically, with zero data loss. RAID-5 will not do that. ZFS can rebuild the array onto a new replacement disk while the old (failing but usable) disk is still working away, providing redundant data to 99.999% of the filesystem. RAID-5 can't. ZFS can do 2 or 3 redundant disks, hot spares, etc. Very cool. And don't even get me started on how cool snapshots are, or partition-less file systems. When sharing zfs file systems to Windows using Opensolaris, snapshots appear as "previous versions" of a disk just like restore points do for NTFS disks. You can use Windows explorer to browse through snapshots and copy deleted or old versions of files from them.

author: gnif
author_rank: Unknown
timestamp: January 21, 2015, 10:37:27 am
content: Quote from: macboy on January 20, 2015, 07:19:15 pmI could not agree more, I just moved from a RAID6 array to RAID-Z, this is the first time I have used ZFS and I must say it is incredible. It also avoids theraid write hole issue. My server only has 8GB of RAM, most of which is dedicated to xen virtual machines leaving only 2GB for the host machine, and ZFS. This performs quite well even before I added the ZIL device.A small note though, making ZFS your root device is rather hard to get right initially, it took me 4-5 hours to figure out how to make Debian happy to use ZFS as it's root file system. I have also used a software (mdadm) RAID1 partition on the disk for /boot, this is because not all versions of GRUB support booting directly from a ZFS volume, should you need to perform recovery it is good to have third party tools such as sysrescuecd to be able to see your /boot volume.

author: JJalling
author_rank: Unknown
timestamp: January 21, 2015, 10:56:22 am
content: Quote from: macboy on January 20, 2015, 07:19:15 pmThat sounds great. The server is mainly for storage, so I will give it a go.Now I'm just waiting for the disks to arriveThanksBR Jonas