Project: TiDB (App Definition and Development / Database)
File: README.md

A table filter is an interface which determines if a table or schema should be accepted for some process or not given its name. This package defines the format allowing users to specify the filter criteria via command line or config files. This package is used by all tools in the TiDB ecosystem. ```go package main import ( "fmt" "github.com/pingcap/tidb/util/table-filter" "github.com/spf13/pflag" ) func main() { args := pflag.StringArrayP("filter", "f", []string{"."}, "table filter") pflag.Parse() f, err := filter.Parse(*args) if err != nil { panic(err) } f = filter.CaseInsensitive(f) tables := []filter.Table{ {Schema: "employees", Name: "employees"}, {Schema: "employees", Name: "departments"}, {Schema: "employees", Name: "dept_manager"}, {Schema: "employees", Name: "dept_emp"}, {Schema: "employees", Name: "titles"}, {Schema: "employees", Name: "salaries"}, {Schema: "AdventureWorks.Person", Name: "Person"}, {Schema: "AdventureWorks.Person", Name: "Password"}, {Schema: "AdventureWorks.Sales", Name: "SalesOrderDetail"}, {Schema: "AdventureWorks.Sales", Name: "SalesOrderHeader"}, {Schema: "AdventureWorks.Production", Name: "WorkOrder"}, {Schema: "AdventureWorks.Production", Name: "WorkOrderRouting"}, {Schema: "AdventureWorks.Production", Name: "ProductPhoto"}, {Schema: "AdventureWorks.Production", Name: "TransactionHistory"}, {Schema: "AdventureWorks.Production", Name: "TransactionHistoryArchive"}, } for _, table := range tables { fmt.Printf("%5v: %v\n", f.MatchTable(table.Schema, table.Name), table) } } ``` Try to run with `./main -f 'employee.' -f '.WorkOrder'` and see the result. The input to the `filter.Parse()` function is a list of table filter rules. Each rule specifies what the fully-qualified name of the table to be accepted. ``` db1.tbl1 db2.tbl2 db3.tbl3 ``` A plain name must only consist of valid [identifier characters] `[0-9a-zA-Z$_\U00000080-\U0010ffff]+`. All other ASCII characters are reserved. Some punctuations have special meanings, described below. Each part of the name can be a wildcard symbol as in [fnmatch(3)]: `` matches zero or more characters `?` matches one character `[a-z]` matches one character between a and z inclusive `[!a-z]` matches one character except a to z. ``` db data.* .backup_ ``` Character here means a Unicode code point, so e.g. U+00E9 () is 1 character. U+0065 U+0301 (e) are 2 characters. U+1F926 U+1F3FF U+200D U+2640 U+FE0F () are 5 characters. Include an `@` at the beginning of the string to specify a file name, which `filter.Parse()` reads every line as filter rules. For example, if a file `config/filter.txt` has content: ``` employees.* *.WorkOrder ``` the following two invocations would be equivalent: ```sh ./main -f '@config/filter.txt' ./main -f 'employees.' -f '.WorkOrder' ``` A filter file cannot further import another file. Leading and trailing white-spaces of every line are trimmed. Blank lines (empty strings) are ignored. A leading `#` marks a comment and is ignored. `#` not at start of line may be considered syntax error. An `!` at the beginning of the line means the pattern after it is used to exclude tables from being processed. This effectively turns the filter into a blocklist. ```ini . !*.Password !employees.salaries ``` Precede any special character by a `\` to turn it into an identifier character. ``` AdventureWorks\.. ``` For simplicity and future compatibility, the following sequences are prohibited: `\` at the end of the line after trimming whitespaces (use `[ ]` to match a literal whitespace at the end). `\` followed by any ASCII alphanumeric character (`[0-9a-zA-Z]`). In particular, C-like escape sequences like `\0`, `\r`, `\n` and `\t` currently are meaningless. Besides `\`, special characters can also be escaped by quoting using `"` or `` ` ``. ``` "AdventureWorks.Person".Person `AdventureWorks.Person`.Password ``` Quoted identifier cannot span multiple lines. It is invalid to partially quote an identifier. ``` "this is "invalid. ``` Use `/` to delimit regular expressions: ``` /^db\d{2,}$/./^tbl\d{2,}$/ ``` These regular expressions use the [Go dialect]. The pattern is matched if the identifier contains a substring matching the regular

For instance, `/b/` matches `db01`. (Note: every `/` in the regex must be escaped as `\/`, including inside `[``]`. You cannot place an unescaped `/` between `\Q``\E`.) When a table name matches none of the rules in the filter list, the default behavior is to ignore such unmatched tables. To build a blocklist, an explicit `.` must be used as the first rule, otherwise all tables will be excluded. ```sh ./main -f '!*.Password' ./main -f '.' -f '!*.Password' ``` In a filter list, if a table name matches multiple patterns, the last match decides the outcome. For instance, given ```ini employees.* !.dep *.departments ``` We get: | Table name | Rule 1 | Rule 2 | Rule 3 | Outcome | |--|--|--|--|| | irrelevant.table | | | | Default (reject) | | employees.employees | | | | Rule 1 (accept) | | employees.dept_emp | | | | Rule 2 (reject) | | employees.departments | | | | Rule 3 (accept) | | else.departments | | | | Rule 3 (accept) | u only create a timer through a client, the timer will not run by default. You must set up a timer runtime to schedule the timers. You can create a new `runtime.TimerGroupRuntime` and then start it, for example: ```go package main import "github.com/pingcap/tidb/timer/api" func main() { ctx := ... // some go context store := ... // Create a timer store. rt := runtime.NewTimerRuntimeBuilder("myGroup", store). RegisterHookFactory("your.hook.class.name", MyTimerHookFactory). SetCond(&api.TimerCond{ Key: api.NewOptionalVal("/"), KeyPrefix: true, }). Build() rt.Start() defer rt.Stop() <-ctx.Done() } ``` We use a builder to build a new timer. You can specify a group name when creating it. The group name is only used by monitoring now. `RegisterHookFactory` registers your custom hook class which will be called when the runtime trys to trigger a event. `SetCond` sets a condition and the runtime will only schedule the timers which match the condition. If you do not set the condition, the runtime will schedule all the timers. If you start more than one timer runtimes with overlapped conditions, the hook functions have a probability to be called more than once. So it's better to use some distributed lock to ensure that only one runtime is running. In the future, we are going to provide some preserved runtimes in TiDB system, so that you can use them directly without creating a new one. You can implement your own hook class by implementing the interface `api.HookClass`, for example: ```go func MyTimerHookFactory(hookClass string, cli api.TimerClient) api.Hook { return &MyTimerHook{ cli: cli, } } type MyTimerHook struct { cli api.TimerClient } func (h *MyTimerHook) Start() { // You should do some init works here. } func (h *MyTimerHook) Stop() { // You should do some clear works here. } func (h *MyTimerHook) OnPreSchedEvent(ctx context.Context, event api.TimerShedEvent) (api.PreSchedEventResult, error) { fmt.Printf("OnPreSchedEvent: %s\n", event.EventID()) return api.PreSchedEventResult{}, nil } func (h *MyTimerHook) OnSchedEvent(ctx context.Context, event api.TimerShedEvent) error { fmt.Printf("OnSchedEvent: %s, event start: %s\n", event.EventID(), event.Timer().EventStart) return h.cli.CloseTimerEvent(ctx, event.Timer().ID, event.EventID()) } ``` The function `MyTimerHookFactory` is used as a parameter of `RegisterHookFactory` in the previous section. It is used to create a new hook instance when the runtime needs it. A parameter `cli` with type `api.TimerClient` will be provided to help to construct the hook, and the hook can use the client to do some further timer operations. All the timers with the same hook class will share the same hook

When the runtime finds a timer that is ready to trigger, it will first find whether the corresponding hook is created. If not, it will call the hook factory to create a new hook instance. Then it will call the hook's `Start` method to do some preparation work. The `Start` method will only be called once. After that, for each trigger action, the runtime will call the hook's `OnPreSchedEvent` method to do some pre-trigger works. `OnPreSchedEvent` returns a `PreSchedEventResult` object to tell the runtime what to do next. In most cases, we just need to return an empty `PreSchedEventResult` to continue the trigger action. But sometimes you may want to delay the schedule for some reason such as some schedule window limits required by business. You can delay the schedule like this: ```go func shouldDelaySchedule() bool { // your function to check whether to delay a timer's schedule } func (h *MyTimerHook) OnPreSchedEvent(ctx context.Context, event api.TimerShedEvent) (api.PreSchedEventResult, error) { if shouldDelaySchedule() { return api.PreSchedEventResult{ Delay: time.Minute, }, nil } return api.PreSchedEventResult{}, nil } ``` In the above example, the delay time is 1 minute. After 1 minute, `OnPreSchedEvent` will be called again, and the trigger action will not continue until the method `OnPreSchedEvent` returns no delay. If you want to carry some extra data to the trigger action, you can set the `PreSchedEventResult`'s `EventData` field. The data will be passed to the trigger action. For example: ```go func (h *MyTimerHook) OnPreSchedEvent(ctx context.Context, event api.TimerShedEvent) (api.PreSchedEventResult, error) { if shouldDelaySchedule() { return api.PreSchedEventResult{ EventData: []byte("mycustomeventdata"), }, nil } return api.PreSchedEventResult{}, nil } ``` The `EventData` will be persisted before the action is actually triggered. So you can use it to do some recovery work if the TiDB process restarts. After `PreSchedEventResult` indicates to continue the trigger action, the event meta including the `EventData` will then be persisted to the store. After that, if you query this timer with the client, you can see the `EventStatus` changes to `TRIGGER` from `IDLE`, and the `EventID` is also not empty. Then the hook method `OnSchedEvent` will be called. You can do some real trigger action here. For example, you can send a message to a message queue or submit some job to an outer job system. If your job is not very heavy, you can even do it directly in this method. Please notice that the method `OnSchedEvent` will only be called once if it returns no error. Only some cases may cause the method `OnSchedEvent` to be called again: The method `OnSchedEvent` returns an error. The runtime restarts and finds a timer's `EventStatus` is `TRIGGER`. You should close the timer event by calling the client's method `CloseTimerEvent` to reset the timer's state to `IDLE`, or the timer will not be able to schedule new events. By default, closing a timer will also reset the `Watermark` field to the value of `EventStart`, but you can specify the next watermark manually. For example: ```go func (h *MyTimerHook) OnSchedEvent(ctx context.Context, event api.TimerShedEvent) error { // do some things ... // close the timer event so that the next event can be scheduled return h.cli.CloseTimerEvent( ctx, event.Timer().ID, event.EventID(), api.WithSetWatermark(time.Now()), // use the current time as the next watermark instead of the event start time ) } ``` Q: What is the difference between the attributes `ID` and `Key` of the timer, they are all unique. A: The key difference is that the `Key` is specified by the user but the `ID` is generated by the

The `ID` is the real identifier of a timer because it is "absolutely" unique because any two timers cannot have the same `ID` even if one timer is deleted. But it is allowed to create a timer with a `Key` that is the same as another timer which deleted before, and it is also allowed to create timers with the same `Key` in different namespaces. So it is better to use `ID` to log events from a certain timer. But `Key` is also useful because it can be meaningful. `Key` can be used to do some semantic works like querying or ensuring the idempotent creation of timers for certain uses. Q: What is the behavior of the timer framework if the TiDB restarts, especially if it shuts down for a long time? A: When the TiDB restarts (which means the timer runtime restarts), the runtime will reload all timers from the store and check their state. If a timer's `EventStatus` is `TRIGGER`, the runtime will try to trigger the event again by calling the hook's method `OnSchedEvent` (`OnPreSchedEvent` will not be called). If the `EventStatus` is `IDLE`, the runtime will try to trigger the event if the timer is time to trigger. Notice that when a runtime shuts down for a long time, there may be many time points to trigger between `Watermark` and the current time, but the hook will still be triggered only once. For example, one timer has a cron expression `0 *` indicating that it should trigger every one hour. If the TiDB shuts down, leave this timer with a watermark `2022-01-01 09:58:00` and then restart at `2022-01-01 12:01:00`, and it tries to trigger the timer. However, there are 3 time points which are not triggered. The hook will still be invoked once with the `EventStart` value `2022-01-01 12:01:00` and `Watmerkark` value `2022-01-01 09:58:00`. It is up to the hook class to determine how to handle this case. For example, it can do the trigger action 3 times in one `OnSchedEvent` call or just do it once and then close the timer with the new watermark `2022-01-01 10:00:00`. For the second case, the timer will be triggered again after a very short time because the next expected trigger time `2022-01-01 11:00:00` is still before the current time. Q: Is it thread-safe for the hook call such as `Start`, `Stop`, `OnPreSchedEvent`, `OnSchedEvent`? A: Yes, it is thread-safe. They are called in one goroutine now, there is no data race without a lock for variables that are only read or written in these methods. But if you create a goroutine manually and access these variables in the background, you should also use a lock to protect them. In the future, the trigger method such as `OnPreSchedEvent` and `OnSchedEvent` may be called in concurrency of different timers, but it will also be guaranteed that these methods for the same timer will be called in sequence and guarantees the happens-before semantics. Q: Will the hook calls from different timers be blocked by each other if some hook calls take a long time? A: Yes, it may happen if two timers share the same hook and one timer's call is slow. The reason is timers that share one hook will be triggered in one goroutine (or maybe a pool with some fixed goroutines in the future). To resolve this issue, you can just do the trigger action background and return a non-error value immediately in the foreground call. Notice that if you do this, you should retry yourself when the trigger action fails.