Shimano Di2 modulation - Page 1
Forum section: electronics / projects

author: Radiohead
author_rank: Contributor
timestamp: September 22, 2020, 09:44:07 am
content: Hello All,I've been busy tearing down a Shimano Di2 eletronic shifter handle to see if I could make my own controller for operating the shifter. Specificly I'm tearing down the E-tube version, so no can-bus interface. The data between the devices is AC coupled onto the DC powerlines that is connected to all other parts of the Di2 system. No transceiver IC or anything, just an dual diode to limit the AC amplitude and some AC coupling between the power input and the microcontroller with a 16MHz oscillator. The C4N part seems to be an 3.3V LDO.PCB:The main frequency is 1MHz and seems to use single period amplitude modulation. Every millisecond the message is repeated and sometimes seems to be acked. The same message is repeated over and over.The reason I've posted this is that I don't recognize the modulation used. I've begun to wonder weither it's not just a waveform created by a DAC. Anyone has any idea what kind of modulation might be used?

author: amspire
author_rank: Super Contributor
timestamp: September 22, 2020, 10:33:03 am
content: Looks like Phase Shift Keying. The phase shifts 180 degrees to indicates the 1's or 0's. Some or all of the first six pulses may be there to establish the phase.I have never used electric derailleurs, but looking at a teardown, it looks like the rear derailleur has a high speed DC motor driving the output shaft through a lot of gearing. So it is not a stepping motor, and there is probably no DAC and linear motion sensor used. There is a magnet involved, so the motor or a geared shaft revolutions may be counted via the magnet and a hall effect sensor. The code may be transmitting a revolution count based on some calibration point. Does the Di2 do any kind of initialisation when it starts such as going to the 1st or 12th gear?Richard

author: amspire
author_rank: Super Contributor
timestamp: September 22, 2020, 11:00:37 am
content: Your best bet may be to look at the Shimano Patents. You can track down the patent numbers here:https://patents.justia.com/patent/10773773.This site does not have the drawings, but with the numbers, you can probably get the patents from Google.

author: Radiohead
author_rank: Contributor
timestamp: September 22, 2020, 12:03:45 pm
content: Thanks amspire/Richard!I had interpreted the change in the signal as some kind of amplitude modulation and not a phase shift, which explains why it didn't match any amplitude modulation I could find. I had compared the scope plots with different modulation waveforms found on the web, but all the drawings of course show no signal aberrations as the do in reality.Do you have any suggestion what I could use to decode/encode the PSK messages? I have an Analog Discovery 2, but there might be something better suited for this task.

author: CoteRotie
author_rank: Contributor
timestamp: September 22, 2020, 04:16:46 pm
content: I'm a Campy only guy, so no help here

author: bson
author_rank: Unknown
timestamp: September 22, 2020, 07:59:36 pm
content: I have two bikes with R8000 Di2, but that's CANbus, and on those versions the derailleurs are smart. The shifters only send up/down commands, and the derailleurs go do it, then transmit the position. The position is picked up by the wireless unit, which sends it to the computer head unit (Garmin Edge 530 or Wahoo Elemnt in my case) which updates its notion of which gear is used. The gear is in terms of 0-(n-1) which is displayed as 1-n. Not sure how full synchro shifting is done. There is no calibration, except Wahoo head units can lose track of which gear you're in - shifting to the extremes will recalibrate it, while the Garmin Edge never loses track even if moved between bikes. So my guess is the Garmins pick up absolute gear indexes, while the Wahoo picks up on the up/down shifts. This is just my WAG based on observing how it works along with some very scant, not very technical information my brain has collected.

author: amspire
author_rank: Super Contributor
timestamp: September 22, 2020, 11:35:12 pm
content: Quote from: Radiohead on September 22, 2020, 12:03:45 pmbson is right. Apparently Shimano has been using CANbus since 2009 for Di2 and since I think Di2 was released in 2009, that means all Di2 should be CANBus. The best bet is probably to get a CAN compatible interface chip or microcontroller.It would be good news. The amount of data needed to send a shift position would only need to be 8 bits, but since it is a premier product, Shimano may go over the top and use 16 bits. It could just be sending the up/down shift commands, but I think I have seen a Shimano Di2 app that allows you to tweak the exact position of each shift. Is this correct? The data packet looks much longer then 16 bits, so I was thinking there is probably some error detecting bits as well that would be hard to work out. CRC bits are a standard part of the CAN protocol, so it solves that issue.Richard.

author: KE5FX
author_rank: Super Contributor
timestamp: September 22, 2020, 11:41:37 pm
content: Crazy that derailleurs are still a thing in 2020. Surely at some point it'll make more sense to use a motor/generator combination to transmit power from the crank to the rear wheel...

author: amspire
author_rank: Super Contributor
timestamp: September 22, 2020, 11:44:55 pm
content: Looks like the hard work may have been done at least for the ANT+ communications:https://hackaday.com/2019/03/26/reverse-engineering-shimano-bike-electronics/

author: amspire
author_rank: Super Contributor
timestamp: September 23, 2020, 02:33:35 am
content: Quote from: KE5FX on September 22, 2020, 11:41:37 pmIf you want cranks, you basically need gears or derailleurs. Derailleurs and cassettes can be very cheap and they work very well.If you want to make a fully electric bike with no pedalling, then you start to need a lot of batteries for a decent range and you will probably have to license the bike as a motor vehicle. At the moment, to get regeneration, you usually need the motor built into the hub of the rear wheel and the current rear wheel motors with regeneration usually do not have enough torque to get an average person up a 20% slope. They also will not have the rating to fully recover the energy to the battery pack when you are going down the same 20% slope. Putting a lot of weight into the rear wheel will affect handling, and to get regeneration with a center mounted motor, you have to eliminate the rear freewheel hub. This will be less efficient when coasting.I do agree that at some point when we have motor assist with regeneration that actually works well and with only a few kilograms of extra weight, then ebikes will probably become the norm. I think it still needs a lot of new technology to get there.

author: Someone
author_rank: Super Contributor
timestamp: September 23, 2020, 06:54:46 am
content: Quote from: KE5FX on September 22, 2020, 11:41:37 pmTough to beat gears when it comes to efficiency.https://www.cyclingabout.com/drivetrain-efficiency-difference-speed-between-1x-2x/Motor-generaotrs have relatively narrow bands of efficient operation, and aren't going to be competitive in any of cost/weight/performance.

author: KE5FX
author_rank: Super Contributor
timestamp: September 23, 2020, 07:01:44 am
content: Quote from: Someone on September 23, 2020, 06:54:46 amBut chain-driven derailleurs are mechanically-annoying objects that require constant maintenance and really only work well the day they come back from the bike shop. Meanwhile, PM motor/generator efficiency can exceed 90%. (Yes, they would both need to be geared appropriately, but those gearsets could be lightweight sealed units.)Note that I'm not asking for batteries, I just don't see why the pedals have to turn a chain drive instead of a generator that drives a motor in the wheel hub.

author: Jeroen3
author_rank: Super Contributor
timestamp: September 23, 2020, 07:55:53 am
content: Quote from: KE5FX on September 22, 2020, 11:41:37 pmThere are variable transmissions with belts on the market for bikes. It's all expensive stuff. Chains are cheap.Above modulation doesn't like like CAN at all. So something is going on.

author: Someone
author_rank: Super Contributor
timestamp: September 23, 2020, 11:42:00 pm
content: Quote from: KE5FX on September 23, 2020, 07:01:44 amYou're doing it wrong if derailleurs aren't working smoothly, sounds like the common complaint from people who have only ever used toy-store/"disposable" quality bikes and haven't ridden something built to last. When do I do maintenance on chain drives? When the chain has stretched and gets replaced, the rest of the time it just works. Want something even more robust? Internal geared hubs and belt drives are available (with a weight penalty).The weight and cost kills an electric transmission, before any issues of efficiency. Efficiency which is not going to be peaking at 90% (citation needed!) over anything more than a tiny part of the operating envelope even for the best technology available (which won't be the lightest). The weight for a chain drive is:https://weightweenies.starbike.com/listings.phpCassette 200gChainrings 150gChain 300gFront Derailleur 100gRear Derailleur 200gShifters 250gSo the parts you can strip off total 1200g, for a transmission that carries peak powers up past 2000W and has very little flex or slip.Swap out to a geared hub and its under 3000g. There is a reason beyond safety that most power steering still has a mechanical linkage between the driver and the road, substituting for a slipping/delayed drive (just as a loose chain would do) kills the connection and feel.P.S. just the hub motor for a typical electric bicycle is 3kg, 3000g, before considering a generator, power electronics, and wiring.Its that sort of thinking that ends up with toy bicycles that weigh 15-20kg but look nice and have all the marketing "check boxes".

author: KE5FX
author_rank: Super Contributor
timestamp: September 24, 2020, 12:56:19 am
content: Sure, this wouldn't result in any sort of racing bike, but it would certainly yield a maintenance-free commuter bike.In my book, if there's any surface with oil and grease that's exposed to the environment, it's a maintenance hassle. I'd be OK if my bike weighed 20% more but never needed maintenance beyond tires, and was always in the right 'gear.'Anyway, back to the original topic, which is making a temperamental mechanical kludge even better by adding what amounts to consumer IoT technology.

author: Someone
author_rank: Super Contributor
timestamp: September 24, 2020, 02:14:42 am
content: Quote from: KE5FX on September 24, 2020, 12:56:19 amIt doesn't meet the requirements of any cycling market, other than your hypothetical "cool tech" in an inappropriate application. Fails in all departments:Price,Weight,Performance,Usability,...with no upside compared to existing options. Even electric bicycles that already have more than half the system in place don't try to put a generator between the pedals.Quote from: KE5FX on September 24, 2020, 12:56:19 amThey exist, IKEA made an attempt at it, and there are a range of utility/commuter bicycles with internal hub gears that are effectively zero customer maintenance. The Netherlands and surrounding countries are full of those everyday bicycles. "Automatic" gearing has been tried several times with little customer acceptance.Electric drivetrain does nothing to address that market, yet electric assist is growing rapidly.Quote from: KE5FX on September 24, 2020, 12:56:19 amc.f. "an exquisitely optimised system refined over a century of continuous use"

author: amspire
author_rank: Super Contributor
timestamp: September 24, 2020, 02:25:06 am
content: Quote from: Jeroen3 on September 23, 2020, 07:55:53 amI think it probably is CAN compatible if the amplitude is 5V p-p rather then 0.5V p-p.CANBus is differential and only one side was captured, so the differential signalling looks a bit like a phase reversal. It looks like a sine wave, but if the CANBus sync causes a sampling at the sine wave peaks, then it would work fine.

author: Jeroen3
author_rank: Super Contributor
timestamp: September 24, 2020, 06:13:44 am
content: CAN bus has bit stuffing after 5 highs. I count 16 highs.

author: Radiohead
author_rank: Contributor
timestamp: September 24, 2020, 06:51:19 am
content: I can find a lot of references which say that the Di2 uses a CAN bus system, but nowhere is it supported by any information.Quote from: amspire on September 22, 2020, 11:35:12 pmI did find the appnote slla336 of TI which describes using a CAN bus with power-over-data. However, I do not find any differential CAN bus signals on the PCB, nor does it seem there's any data on the negative supply wire. Only on the positive supply input is there AC decoupling and a diode voltage clamp with the dual package diode. If there was differential signalling, I would expect the same kind of input protection on the negative input. The capacitors that are of the same size as the one next to the dual package diode are all connected to the 3.3V or 1.8V supplies. Only the diode under the mounting holes functionality is unclear. Might just be some reset pin circuitry. There's no communication signal on it and on both pads I measure 3.3V.Quote from: Jeroen3 on September 23, 2020, 07:55:53 amI guess it's a bit of both. according:https://www.bicycles.net.au/forums/viewtopic.php?t=84327&start=36QuoteThere are some single wire CAN bus IC's like the MC33897 for example, but I would be surprised to find that in a micro controller as a peripheral.Quote from: amspire on September 24, 2020, 02:25:06 amMy probe and oscilloscope are both on x1.0, so the amplitude is not 5V. The TI app note also discusses the problem with AC coupled CAN bus, where due to the AC decoupling the differential amplitude is not large enough for the receiver to reach the thresholds. In order to fix this they propose a solution with two CAN bus transceivers which work together in order to transmit the same data and are connected invertedly on the CAN bus in order to create the necessary signal amplitude.Thanks for all your replies so far!

author: bson
author_rank: Unknown
timestamp: September 25, 2020, 02:41:41 am
content: Quote from: Jeroen3 on September 23, 2020, 07:55:53 amYeah, that's exactly what I was thinking... Definitely looks like PSK. But it could be the shifters just pulse at some rate and don't really talk CANbus at all.It does seem like early Dura-Ace Di2 was maybe 4-wire?

author: amspire
author_rank: Super Contributor
timestamp: September 26, 2020, 11:21:43 am
content: Quote from: bson on September 25, 2020, 02:41:41 amThe initial Di2 was 5 wire and I think they then went to 4 wire. These had power and data separated The newest Di2 now have 2 wire with the wires carrying both power and data simultaneously.Apparently they have taken CANBus but have used their own communications protocols. Seems like they have also modified CAN voltage levels and waveforms as well.Shimano has done a really terrible job keeping compatibility even between its own Di2 models. The derailleurs are basically very simple devices, and it would have been pretty trivial at keeping compatibility, even if it meant cable converters to go from 5 to 2 wire. They also do not let you use an 11 speed Di2 derailleur with a 10 speed rear cassette, even though the 11 speed derailleur should be able to get the correct positions for all the 10 speed gears.

author: bson
author_rank: Unknown
timestamp: September 26, 2020, 07:18:38 pm
content: Di2 moved the indexing from the shifters to the derailleurs, so yeah if you want to change your indexing you need to replace the rear derailleur. And since the derailleurs talk they need to be mutually compatible and you need to replace the front to match the rear. But you can keep your shifters. This is different from mechanical where indexing was in the shifters and the derailleurs only had to implement a specific pull ratio and range of motion.

author: Radiohead
author_rank: Contributor
timestamp: October 02, 2020, 01:33:54 pm
content: Hello all,I've gotten a bit further, but in order to determine the Power Line Communication protocol I woud like to digitize the analog signal in an automated repeatable manner. The attached csv file was created with an oscilloscope.The attached csv(changed the extension to .txt in order to upload on the forum) file shows the communication to the connection hub in order to read its firmware version with a command and two replies. I know the firmware version is 3.0.4, which is possibly transmitted as 0x3004 or a variation of. Both replies 1 and 2 are different, so there might be some sort of packet counter and with maybe a 2's complement 8-bit checksum. In different messages I got quite long periods of a continuous wave-form, so I think bit-stuffing might not be used or limited.I've installed Phyton, but have only a limited experience with Matlab (and currently no license or active trial) and all most no experience with Phyton. Is there something, a Phyton library or module etc anyone can recommend to read the following csv file and digitize the following wave forms? I would like to create a binary/hexadecimal output from the wave forms according to different interpretations of what the different waveforms might represent bit-wise. Then it will be a matter of trying to interpret the binary data with some expected values.Thanks in advance!

author: qosch
author_rank: Newbie
timestamp: March 15, 2021, 11:18:51 am
content: Hello Radiohead,have you gotten any further than this? I've also taken (=dremeled) apart an XTR trigger shifter in hope they'd have used an IC that I could just buy to talk to the shifter but no luck. Concerning the modulation I am afraid I can't contribute anything useful. Since I only want to receive the button presses from the shifter I thought a brute force method could be to digitize the AC coupled signal on the other side and see if it matches a reference signal (assuming the same button will always trigger the same signal waveform).Cheers

author: Teknotronix
author_rank: Regular Contributor
timestamp: August 14, 2021, 12:06:57 am
content: Perhaps this transceiver would allow us to interface with the Di2 version of CAN?https://yamar.com/product/dcan500/Eval board:https://yamar.com/product/dcan500-evb/

author: mark.s
author_rank: Newbie
timestamp: October 20, 2021, 08:23:39 am
content: Yes, it is binary phase shifting with IF 1MHz and data rate at 500kbps. Data and carrier frequency are synced so it is quite easy to sample the data. I've written a simple octave file that does the demodulation, prints it out and shows the sample points in a plot.Here is "repeat1", 0001100001000000001111110Code:import = csvread ("plc.txt");import(:,1) = [];all = import;repeat1 = import(6000:7100);repeat2 = import(51000:54000);repeat3 = import(77600:78800);repeat4 = import(103000:104650);cmd = import(27500:30400);reply1 = import(61000:64200);reply2 = import(86950:89800);ack1 = import(30500:31800);ack2 = import(64300:65600);ack3 = import(89800:91800);signal = cmd;threshold = 0.15;sample_th = 0.05;plot(signal)receive = 0;peakdetect = 0;oldpeak = 0;sync = 0;threshold = 0.15;data = zeros(1);sampletime = zeros(1);bitcount=1;for i = [1:length(signal)]# got threshold valueif (receive == 0 && signal(i) > threshold)peakdetect = 1;endif# finding peakif (peakdetect == 1 && signal(i) < oldpeak)peakdetect = 0;receive = 1;#Use next half wave as sample point. -1 for maximumsync = i+10-1;endif# receivingif (receive == 1 && rem(i-sync,40) == 0)sampletime(i)=0.1;if (signal(i) > sample_th)data(bitcount)=1;bitcount = bitcount+1;elseif (signal(i) < -sample_th)data(bitcount)=0;bitcount = bitcount+1;elsereceive=0;datadata = zeros(1);bitcount =1;endifelsesampletime(i)=0;endifoldpeak = signal(i);endforhold on;plot(sampletime);hold off;

author: Radiohead
author_rank: Contributor
timestamp: October 20, 2021, 04:23:45 pm
content: Thanks Mark.s!I had given up on this project as I didn't get any further with it. I do not know whether I'll pick it back up as I've got my hands full for the foreseeable future. However it's still great to see that someone figures out the modulation type.

author: avandalen
author_rank: Contributor
timestamp: July 26, 2022, 07:54:59 am
content: Shimano Di2 SWCAN hackI have developed my own electronic system to control an e-bike:https://www.avdweb.nl/solar-bike/electronics/robust-two-wire-gnd-5v-current-controlled-bus-for-ebikesI want to use the electronic shifting Alfine Di2 11-speed hub with the motor unit MU-UR500. But I don’t use any other Shimano parts, such as the Di2 battery, switch, junction box or display.So, I have to control the motor unit by software myself. No one has hacked the Di2 system till now, so that's what I want to do.Shimano Di2 uses its own, non-standard, CAN variant (probably SWCAN) in which the power supply and the CAN signal go together.Who can lend me a complete system so I can hack the Di2?

author: Henrikw
author_rank: Newbie
timestamp: November 17, 2023, 10:11:49 pm
content: @avandalen did you get any further on this. Im interested on the same thing.

author: c0d3z3r0
author_rank: Regular Contributor
timestamp: June 28, 2024, 08:54:12 am
content: I got a Shimano SW-E6000 switch here, which I'd like to use for my custom eBike controller.Ofc I could just get rid of the MCU inside and connect the buttons directly, but where's the fun then, eh?What I know so far:- The packaging says Di2, that's how I found this thread.- Inside there's a 3.3V LDO and a MCU: 11XW marking- The circuit looks very similiar to the SW-S705, just smaller package sizes- I was able to power it with 5V and could see some weird AC waveforms on the oscilloscope, but they look different like Radiohead's- Shimano references CAN bus a few times, e.g.https://bike.shimano.com/en-EU/technologies/component/details/steps/drive-unit.html- There are two protocols, the older ACC and the newer Di2.- Shimano support said StePs supports both protocols, that's why old and new components work. (Can't find the blog post anymore, but- Since there's only two wires, Di2 might be SWCAN (that's what people say in other forums as well).- OTOH there seem to be different protocols among Di2 components. There's a comment mentioning timed pulseshttps://carltonbale.com/shimano-di2-everything-you-need-to-know/comment-page-5/#canbus_protocolI found this post in the German pedelec forum:https://www.pedelecforum.de/forum/index.php?threads/optimierung-von-shimano-steps-systemen-mit-den-emax-softwarewerkzeugen.51623/page-178#post-1596705- That guy claims they have decapped, reverse engineered that chip:- It's a custom chip based on Renesas RX- The protocol is not "just simple CAN" but a "very special" proprietary protocol- At least that sounds like it is somehow similiar to CAN - maybe some modified SWCAN that can be implemented without dedicated transceiver?This video confirms @mark.s's results.https://youtu.be/Dyn7XHsTmgI?t=530Note beside: that guy is claiming there is no way that Di2 could be CAN + power over a single wire, but he's only talking about regular two-wire CAN, but completely ignoring SW-CAN and possible proprietary modifications of it. Also there ist DCAN500, which is CAN-over-Powerline.I found reddit posts from people saying that they were able to simply use regular CAN modules (like mcp2515) on SWCAN busses. Might be worth a try :O@Radiohead: Can you give some more details on how you powered your SW-S705 and connected your osci?

author: c0d3z3r0
author_rank: Regular Contributor
timestamp: June 28, 2024, 08:24:14 pm
content: I created schematics for the SW-E6000.SW-E6000.pdf(112.88 kB - downloaded 102 times.)Also, here are some of my measurements. I'm now relatively confident that this is PSK as well, but it still looks weird. Even when I attach to TP4 :/Power line:TP4:

author: Radiohead
author_rank: Contributor
timestamp: June 29, 2024, 12:19:12 pm
content: QuoteI probed the circuit and found a circuit point where the AC signal of the supply was coupled onto a DC bias before entering the MCU. I used a complete system with the original battery which includes an inductor to provide a high impedance so the signal can be injected on the DC power supply. If you use a power supply a series inductor is required or the impedance will be to low to function.QuoteI'm not familiar with CAN-bus systems and I returned the Di2 system to it's owner so Ican't help you any further. I had bought a cheap rtlsdr to take an attempt at decoding the signal but never got GNUradio working.

author: c0d3z3r0
author_rank: Regular Contributor
timestamp: July 04, 2024, 05:54:38 pm
content: Quote from: Radiohead on June 29, 2024, 12:19:12 pmAh well, there is one on the board, but maybe it's not enough

author: SteveThackery
author_rank: Super Contributor
timestamp: July 04, 2024, 06:27:27 pm
content: Quote from: KE5FX on September 22, 2020, 11:41:37 pmCan't get close to the efficiency of sprockets and a chain. Every percent counts on a bike.You might have heard of the hydraulic system which was experimented with some years ago. The pedals drove a hydraulic pump, the fluid from which then drove a hydraulic motor in the rear hub. Lots of advantages: the gear ratios can be implemented inside the hydraulic motor, so no external complications; the whole kit is clean, maintenance free, compact, etc. It failed for one simple reason: poor efficiency compared with chain and sprockets.I totally share your bogglement that derailleurs are still a thing. I would describe them as an exceptionally well developed bad idea. The idea of requiring extra-flexible chains so they will tolerate running on sprockets well out of line with each other shouts bodgery of the first order. And the lengths they go to, with bent teeth to pick up the chain and drag it across the sprockets... UGH! It's horrible, horrible.AND YET - when they are set up properly they work brilliantly, far better than they really should. 😄