PCIe

# link

https://pc.watch.impress.co.jp/docs/2009/0310/interface01.htm

https://pc.watch.impress.co.jp/docs/2009/0318/interface04.htm

# Restart

https://bugzilla.kernel.org/show_bug.cgi?id=209395#c1
```
A pci generic sequence to try to restart things up might look like:

  echo 1 > /sys/bus/pci/devices/0000:03:00.0/reset
  echo 1 > /sys/bus/pci/devices/0000:03:00.0/remove
  echo 1 > /sys/bus/pci/rescan
```

# MPS

https://blog.linuxplumbersconf.org/2017/ocw/system/presentations/4737/original/mps.pdf

```
tokunori@DESKTOP-ESV85JT:~/linux$ git grep MPS drivers/pci
drivers/pci/Kconfig:      MPS (Max Payload Size) and MRRS (Max Read Request Size) are PCIe
drivers/pci/Kconfig:      The following choices set the MPS and MRRS optimization strategy
drivers/pci/Kconfig:      Use the BIOS defaults; don't touch MPS at all.  This is the same
drivers/pci/Kconfig:      Default choice; ensure that the MPS matches upstream bridge.
drivers/pci/Kconfig:      Use largest MPS that boot-time devices support.  If you have a
drivers/pci/Kconfig:      will use the largest MPS that's supported by all devices.  This
drivers/pci/Kconfig:      Use MPS and MRRS for best performance.  Ensure that a given
drivers/pci/Kconfig:      device's MPS is no larger than its parent MPS, which allows us to
drivers/pci/Kconfig:      keep all switches/bridges to the max MPS supported by their
drivers/pci/Kconfig:      Set MPS = 128 for all devices.  MPS configuration effected by the
drivers/pci/Kconfig:      other options could cause the MPS on one root port to be
drivers/pci/Kconfig:      different than that of the MPS on another, which may cause
drivers/pci/Kconfig:      hot-added devices or peer-to-peer DMA to fail.  Set MPS to the
drivers/pci/controller/pcie-brcmstb.c:#define  PCIE_MISC_MISC_CTRL_PCIE_RCB_MPS_MODE_MASK       0x400
drivers/pci/controller/pcie-brcmstb.c:   * RCB_MPS_MODE, RCB_64B_MODE
drivers/pci/controller/pcie-brcmstb.c:  u32p_replace_bits(&tmp, 1, PCIE_MISC_MISC_CTRL_PCIE_RCB_MPS_MODE_MASK);
drivers/pci/controller/pcie-iproc.c:     * MPSS is not being set properly (as it is currently 0).  This is
drivers/pci/controller/pcie-iproc.c:     * is not modifiable by firmware.  Set this to 2 (e.g., 512 byte MPS)
drivers/pci/controller/pcie-iproc.c:     * so that the MPS can be set to the real max value.
drivers/pci/controller/pcie-rcar-ep.c:  /* device supports fixed 128 bytes MPSS */
drivers/pci/controller/pcie-rockchip-host.c:    status &= ~PCIE_RC_CONFIG_DCSR_MPS_MASK;
drivers/pci/controller/pcie-rockchip-host.c:    status |= PCIE_RC_CONFIG_DCSR_MPS_256;
drivers/pci/controller/pcie-rockchip.h:#define   PCIE_RC_CONFIG_DCSR_MPS_MASK           GENMASK(7, 5)
drivers/pci/controller/pcie-rockchip.h:#define   PCIE_RC_CONFIG_DCSR_MPS_256            (0x1 << 5)
drivers/pci/hotplug/cpqphp_core.c:              dbg("System seems to be configured for Full Table Mapped MPS mode\n");
drivers/pci/hotplug/ibmphp_core.c:static int irqs[16];    /* PIC mode IRQs we're using so far (in case MPS
drivers/pci/pci-acpi.c:  * Don't allow _HPX to change MPS or MRRS settings.  We manage
drivers/pci/pci.c:/* PCIe MPS/MRRS strategy; can be overridden by kernel command-line param */
drivers/pci/pcie/dpc.c: pci_read_config_dword(pdev, cap + PCI_EXP_DPC_RP_PIO_IMPSPEC_LOG, &log);
```

```
tokunori@DESKTOP-ESV85JT:~/linux$ git grep MPS drivers/nvme/host/
drivers/nvme/host/core.c:       dev_page_min = NVME_CAP_MPSMIN(ctrl->cap) + 12;
drivers/nvme/host/core.c:       ctrl->ctrl_config |= (NVME_CTRL_PAGE_SHIFT - 12) << NVME_CC_MPS_SHIFT;
drivers/nvme/host/core.c:       u32 page_shift = NVME_CAP_MPSMIN(ctrl->cap) + 12, val;
```

http://billauer.co.il/blog/2011/05/pcie-pci-express-linux-max-payload-size-configuration-capabilities-tlp-lspci/

https://docs.xilinx.com/r/en-US/pg302-qdma/Max-Payload-Size-for-PCIe-Max-Read-Request-Size

```
Max Payload Size for PCIe, Max Read Request Size

When the master bridge receives a SLVERR response from the addressed AXI slave, the request is discarded and the Master SLVERR (MSE) interrupt is asserted. If the request was non-posted, a completion packet with the Completion Status = Completer Abort (CA) is returned on the bus for PCIe.
```

```
PCIe の最大ペイロード サイズ、最大読み取り要求サイズ

マスター ブリッジがアドレス指定された AXI スレーブから SLVERR 応答を受信すると、要求は破棄され、マスター SLVERR (MSE) 割り込みがアサートされます。 要求がポストされていない場合、Completion Status = Completer Abort (CA) の完了パケットが PCIe のバスに返されます。
```

```
Completion Packets

When the MAX_READ_REQUEST_SIZE is greater than the MAX_PAYLOAD_SIZE, a read request for PCIe can ask for more data than the master bridge can insert into a single completion packet. When this situation occurs, multiple completion packets are generated up to MAX_PAYLOAD_SIZE, with the Read Completion Boundary (RCB) observed.
```

## Kernel command line parameter

https://www.kernel.org/doc/html/v5.4/admin-guide/kernel-parameters.html

```
                pcie_bus_tune_off       Disable PCIe MPS (Max Payload Size)
                                tuning and use the BIOS-configured MPS defaults.
```

https://blog.linuxplumbersconf.org/2017/ocw/system/presentations/4737/original/mps.pdf

```
pcie_bus_perf Set device MPS to the largest allowable MPS
based on its parent bus. Also set MRRS (Max
Read Request Size) to the largest supported
value (no larger than the MPS that the device
or bus can support) for best performance.
```


## BIOS

https://www.techarp.com/bios-guide/pci-e-maximum-payload-size/

https://www.supermicro.com/support/faqs/faq.cfm?faq=22377

https://www.evga.com/support/faq/FAQdetails.aspx?faqid=58420

https://support.huawei.com/enterprise/en/doc/EDOC1100134745/a21237cf/pci-express-configuration

https://www.digikey.in/htmldatasheets/production/2421822/0/0/1/som-5991-user-manual.html

## lspci

```
tokunori@tokunori-desktop:~$ sudo lspci -vvv -d ::0108 -nn
0d:00.0 Non-Volatile memory controller [0108]: Intel Corporation SSD 660P Series [8086:f1a8] (rev 03) (prog-if 02 [NVM Express])
        Subsystem: Intel Corporation SSD 660P Series [8086:390d]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 69
        NUMA node: 0
        Region 0: Memory at fcb00000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x4 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [b0] MSI-X: Enable+ Count=16 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00002100
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [158 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [178 v1] Latency Tolerance Reporting
                Max snoop latency: 1048576ns
                Max no snoop latency: 1048576ns
        Capabilities: [180 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=32768ns
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: nvme
        Kernel modules: nvme

```

https://atmarkit.itmedia.co.jp/ait/articles/1901/18/news046.html

### tree

https://eng-entrance.com/linux-command-lspci

```
tokunori@tokunori-desktop:~$ lspci -t
-[0000:00]-+-00.0
           +-01.0
           +-01.2-[01-0b]----00.0-[02-0b]--+-03.0-[03-08]----00.0-[04-08]--+-01.0-[05]----00.0
           |                               |                               +-03.0-[06]--
           |                               |                               +-05.0-[07]----00.0
           |                               |                               \-07.0-[08]--
           |                               +-08.0-[09]--+-00.0
           |                               |            +-00.1
           |                               |            \-00.3
           |                               +-09.0-[0a]----00.0
           |                               \-0a.0-[0b]----00.0
           +-02.0
           +-03.0
           +-03.1-[0c]--+-00.0
           |            \-00.1
           +-03.2-[0d]----00.0
           +-04.0
           +-05.0
           +-07.0
           +-07.1-[0e]----00.0
           +-08.0
           +-08.1-[0f]--+-00.0
           |            +-00.1
           |            +-00.3
           |            \-00.4
           +-14.0
           +-14.3
           +-18.0
           +-18.1
           +-18.2
           +-18.3
           +-18.4
           +-18.5
           +-18.6
           \-18.7
```

```
tokunori@tokunori-desktop:~$ sudo lspci -vvv -s 00:03.2
00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin ? routed to IRQ 28
        Bus: primary=00, secondary=0d, subordinate=0d, sec-latency=0
        I/O behind bridge: 0000f000-00000fff [disabled]
        Memory behind bridge: fcb00000-fcbfffff [size=1M]
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0
                        ExtTag+ RBE+
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (downgraded), Width x4 (downgraded)
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #0, PowerLimit 75.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState+
                RootCap: CRSVisible+
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- LN System CLS Not Supported, TPHComp+ ExtTPHComp- ARIFwd-
                         AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS-
                DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- LTR+ OBFF Disabled, ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee0a000  Data: 0021
        Capabilities: [c0] Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
        Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn+ NFERptEn+ FERptEn+
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [270 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [2a0 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Capabilities: [370 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                L1SubCtl2:
        Capabilities: [3c4 v1] Designated Vendor-Specific: Vendor=1022 ID=0001 Rev=1 Len=44 <?>
        Capabilities: [400 v1] Data Link Feature <?>
        Capabilities: [410 v1] Physical Layer 16.0 GT/s <?>
        Capabilities: [440 v1] Lane Margining at the Receiver <?>
        Kernel driver in use: pcieport

```

```
tokunori@tokunori-desktop:~$ sudo lspci -vvv -d :1480:600 -nn 
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

```